In this paper, we investigate whether Software Development Effort Estimations (SDEEs) predictions can be improved using commonly used machine learning algorithms such as Linear Regression, Decision Tree Regression, Random Forest Regression, XGBoost Regression, CatBoost Regression, and LightGBM Regression. To prevent the data leakage and enhance the TAWOS agile open-source software project dataset using Tabular Variational Autoencoder (TVAE) and Truncation Normal Data distribution we also apply additional scaling. Hyperparameter optimization with Optuna was conducted on 21 model-data combinations based on 5-fold crossvalidated adjusted R², mean squared prediction error (MSPE), and Pearson's correlation coefficient. The Random Forest Regressor trained on TVAE-augmented data achieved the best results, with an adjusted R² of 0.59, a Pearson's correlation of 0.81, and an MSPE of 140011, indicating strong predictive accuracy. The CatBoost Regressor on regular data ranked second, with an adjusted R² of 0.39, a Pearson's correlation of 0.74, and an MSPE of 200011. The Decision Tree Regressor, despite a high training correlation, performed the worst, with an adjusted R² of 0.35, a Pearson's correlation of 0.76, and an MSPE of 234500, indicating weaker performance. Ultimately, we aimed to reduce the gap between expected and actual software development efforts, thereby minimizing associated risks. The results of this study can significantly enhance software development project planning and management.
Designing and developing software requires both high-quality data and high accuracy to ensure the
overall success of the project. Consequently, Software Development Effort Estimation (SDEE) is vital
in project management, determining project feasibility, and impacting the distribution of funds [
This paper aims to answer to what extent the models CatBoost, XGBoost, RandomForest, LightGBM, and Decision Trees can, in comparison to the Linear Regression baseline model, accurately estimate the story points required for SDEE within the TAWOS dataset. To further elaborate on speciEc aspects of model specialization, we will also seek to compare the machine learning models’ abilities to estimate story points using diFerent evaluation methods such as Pearson’s correlation, Mean Squared Prediction Error (MSPE), and adjusted R2. We will examine how hyperparameter tuning aFects these metrics and if data augmentatio n techniques such as min-max scaling, transformation to a truncated normal distribution, and the use of a Tabular Variational Autoencoder (TVAE) can optimize machine learning models and enhance their accuracy. Therefore, the research contributions (RCs), along with their underlying motivations, are as follows:
Main RC: To what extent can additional regression models, in comparison to the baseline model, accurately estimate the story points required for SDEE within the TAWOS dataset? Sub-RC1: How do regression models compare in their ability to estimate the story points required for soDware development within the TAWOS dataset, considering evaluation metrics such as Pearson’s correlation, MSPE, and adjusted R2? Sub-RC2: How can multiple data augmentation techniques, including min- max scaling, transformation to a truncated normal distribution, and the use of a TVAE, be employed to optimize regression models for enhanced prediction accuracy of actual eFort in SDEE? Sub-RC3: How does hyperparameter tuning inHuence the performance of regression models in the context of SDEE within the TAWOS dataset?
The rest of the paper is organized as follows: Section 1 provides an overview of the current stateof-the-art literature in the Eeld of soDware project estimation, focusing on machine learning and deep learning methods, and the poor use of any data augmentation techniques. Section 2 presents the methodology pipeline. Section 3 discusses the research Endings. Section 4 delves into a detailed discussion of the obtained results. Concluding remarks, along with limitations and future directions, are provided in Section 5.
Historically, numerous software projects either failed or were left unfinished due to inadequate
processes. Commonly employed methods included similarity-based estimation, the analysis and
synthesis method, expert knowledge-based estimation, and various parametric techniques [
Recent advancements in machine learning, also increased the popularity of using deep learning in
software estimation field [
To summarize, reducing model complexity while maintaining accuracy is a desirable state of each software industry use case. State-of-the-art models can offer a more efficient and effective approach to modeling complex systems. Data augmentation techniques help us continue performing analysis and understanding outcomes by providing more accurate and straightforward models.
In this section, we will describe the steps for conducting the experimental part of our research. An illustration of the methodology pipeline is given in Figure 1. The TAWOS dataset comprises 31960 issues from 26 projects from repositories such as Atlassian, Apache, Appcelerator, Hyperledger, MongoDB, Sonatype, Moodle, Talendforge, and similar sources, where diFerent programming languages such as Java, Python, C#, Go and others were used for diFerent projects. It oFers detailed information on versioning, issue tracking, developer assignments, and resolution times. This data is invaluable for research in soDware testing, maintenance, and task optimization. It includes version details like name, description, release date, and status (archived or released) and tracks issues through aFected and Ex versions. Developer assignment data helps in recommending the best developer for new issues and optimizing task assignments based on work-load. The dataset also provides issue status transitions, enabling the analysis of bug Ex times and triage, thereby supporting advanced research using machine learning models.
While #Components and #Developers are useful for detecting the anomalies in speciEc components produced for each user story from diFere nt developer, #Change Log is likely to be more directly correlated with the eFort involved in the project (as reHected in story points) because it captures the actual work being done in terms of changes and updates to the project. This creates a fuller picture when combined with #Issues and #Bugs. Finally, the distribution of the target variable Story Points can be seen in Figure 2, along with descriptive statistics of the input features in Table 1. The smooth line shows the probability density, indicating how story points are distributed in the dataset. This helps to see the trend and pattern of the target variable distribution. "Target Value" on the x-axis represents the values of story points. "Frequency" on the y-axis shows how many times a particular target value appears in the dataset. The quantiEcation step represents the width of the interval for grouping story points. Story points are grouped into bins (intervals) to better display the data distribution. This visualization helps to see how the values are distributed in speciEc ranges. The graph shows that there were about 40 projects in the Erst bin interval, but this does not mean that there were projects with exactly 0 story points. The interval from 0 to 500 can include projects with story points between 0 and 500. The minimum mark on the x-axis is 0 for histogram visualization, which oDen starts at 0 for clearer representation and interpretation of the distribution. Therefore, the x-axis mark does not mean that there were projects with exactly 0 story points.
The Table 1. presents the descriptive statistics for the chosen TAWOS dataset [
bugs.
Defects or problems in the software that need fixing.
Record of all notable changes made to the project.
Measure of effort required to complete a user story or task, aiding in
estimation and planning.
Most projects require relatively low effort. However, there are a few projects that demand significantly higher effort, resulting in a long tail to the right. The data was split using an 80/20 Pareto Split, out of which 80% of the data was used for training and the remaining 20% of the data for the final testing of the models on unseen data. Additionally, it was experimented with 60/20/20 ratio, but better results were observed with 80/20 after numerous trial-error attempts. Moreover, the models were evaluated using 10-fold cross-validation, where the dataset is split into 10 parts, each serving as a test set while the rest are used for training, repeated 10 times. This mitigates the risk of relying on a single split and provides a reliable performance estimate. Additionally, statistical significance tests confirmed that performance differences were not due to random chance, ensuring robust evaluation. The whole data set was scaled using the Min-Max algorithm in order to have a uniform scale for all features and create more homogenous nature of the dataset. Data augmentation techniques such as Tabular Variational AutoEncoder (TVAE) and Truncated Normal Distribution were implemented. Truncated Normal Distribution is a statistical technique used to enlarge the dataset by creating new data points within a specific range derived from the normal distribution, again tailoring the statistical characteristics of the data from the original dataset. TVAE is a type of generative model tailored specifically for creating synthetic data. By these adjustments three different datasets were created, namely, scaled, TVAE and Truncated, to see which performed the best. In this research, we used six different machine learning models and trained them on each dataset variant. These models were XGBoost Regression, Random Forest Regression, CatBoost Regression, Decision Trees Regression, and LightGBM Regression. Linear Regression was also included as a baseline model to compare with the other models, totaling six models. To further increase the accuracy of the model's predictions we conducted hyperparameter optimization using Optuna [20]. Evaluation metrics such as Mean Squared Prediction Error (MSPE), adjusted R2, and Pearson's correlation coefficient were implemented with 5fold cross-validation, resulting in three model/dataset combinations with the most optimal performance.
In this section, we will delve into the details of the results obtained by each model and compare the overall results. We recorded the training time for each model and calculated the average. The average training times are presented in Table 3. Three models were identiEed as the best performers, each excelling in a diFerent metric. The performance of these models on the test set is presented in Table 4 and Figure 7. The hyperparameters for the Enal models are detailed in Table 5. As indicated in the tables and Egure, these models are labeled 1 through 3 and are further described below. The residuals for all three models in Figure 5, while showing some increase, suggest that each model captures the underlying trend reasonably well, with Model 3 being the most accurate. Model 3's smaller residuals indicate it predicts Story Points with greater precision, making it the best performing model among the three. Despite some areas for improvement, all models demonstrate their ability to follow the data's trend and provide useful predictions. l e d o
M
Regressor CatBoost Regressor Decision Tree
Regressor itc led tehn scad
Sy an
2R . j d A
Model 1 was trained on TVAE-augmented data and achieved the best performance among all models. It was evaluated based on the 5-fold cross-validated average adjusted R2 (0.55) on its training data. On the final 20% test set, it maintained strong performance, achieving an adjusted R2 of 0.59, a Pearson’s correlation coefficient of 0.81, and an MSPE of 140001. These results indicate a good fit, strong alignment with observed data, and enhanced prediction accuracy. 4.2.
Model 2 was trained on unaugmented data and achieved the second-highest performance across the three test metrics on the final 20% split. Based on the 5-fold cross-validated average MSPE (328288) on its training data, it obtained an adjusted R2 of 0.39 and a Pearson’s correlation coefficient of 0.74 on the final test data. These metrics indicate a weak goodness-of-fit and moderate alignment with the observed data. Additionally, its lower average MSPE of 200011 suggests moderate prediction accuracy.
Model 3 was trained on the original data and exhibited the weakest performance among the three models on the final 20% test split, despite achieving a high 5-fold cross-validated average Pearson’s correlation coefficient during training (0.77). On the test set, the model resulted in a poor adjusted R2 of 0.35 and a Pearson’s correlation coefficient of 0.66, indicating weak goodness-of-fit and poor alignment with the observed data. Furthermore, its average MSPE of 234500 suggests moderate prediction accuracy. Consistent with our hypothesis, the baseline linear regression model is consistently outperformed by all other models across all evaluation metrics (adjusted R2, MSPE, and Pearson’s correlation coefficient). This is evidenced by the top-performing models being the Random Forest Regressor (Model 1), CatBoost Regressor (Model 2), and Decision Tree Regressor (Model 3). These models co-incidentally follow a clear order of efficacy as shown in Figure 6 and Figure 7. Our findings also suggest that data augmentation can be beneficial for the TAWOS dataset. Model 1, trained on the TVAE-augmented data, achieved the best performance on the final test set for all three metrics, indicating the potential effectiveness of TVAE in improving performance on the TAWOS dataset. Examining the predicted data, as well as the residuals, revealed a trend where the models exhibit greater accuracy in predicting software projects with lower actual effort values. Conversely, projects with higher actual effort proved more challenging to predict. This phenomenon can be also seen in Figure 5.
The results highlight that advanced machine learning models outperform the baseline linear regression model across the performance metrics. Specifically, the Random Forest Regression (Model 1), CatBoost Regressor (Model 2), and Decision Tree Regressor (Model 3) emerged as the bestperforming models, each having its unique strengths and weaknesses, provoking further discussion. An important aspect of our study was analyzing the correlation between input features and the target variable, Story Point. The correlation heatmap indicated strong positive correlations between several key variables (scaling factor #Issues and #Bugs at 0.60, #Bugs and #Change Log with Story Points at 0.39 and 0.47). The Random Forest Regression model (Model 1), based on a 5-fold cross-validated average, achieved the highest adjusted R2 score of 0.55 when trained on the TVAE-augmented dataset. This result underscores the model’s ability to handle complex data environments by effectively capturing underlying data patterns. The data augmentation through TVAE appears to have provided a more detailed dataset that allowed the Random Forest model to make more accurate predictions. However, this also raises questions about the computational complexity and the potential for overfitting, based on the high-dimensional nature of the augmented data itself. The final evaluation on the 20% data split showed an adjusted R2 of 0.59, a Pearson’s correlation coefficient of 0.81, and an MSPE of 140011, indicating strong performance The CatBoost Regression model (Model 2) performed well in minimizing the Mean Squared Prediction Error (MSPE) with a value of 328288 on the scaled dataset. This highlights its potential for practical applications in software project management and suggests that it can effectively handle imbalanced data distributions and various feature scales, making it a versatile model. However, its relatively lower adjusted R2 score may indicate that it may not fully capture the variability in the data, pointing to potential areas for improvement in feature selection and model tuning. The final evaluation revealed an adjusted R2 of 0.39, a Pearson’s correlation coefficient of 0.74, and an MSPE of 200011, reflecting moderate predictive accuracy but highlighting areas for improvement. Decision Tree Regressor (Model 3) demonstrated a strong linear relationship between predicted and actual values with a correlation coefficient of 0.66. However, its lower adjusted R2 score of 0.35 and MSPE of 234500 suggest potential overfitting scenarios, where the model fits the training data but fails to generalize to new data. The aforementioned final evaluation on the 20% data split indicates that it might benefit from more sophisticated ensemble methods to potentially improve the algorithm’s generalization capabilities. Despite lower adjusted R2 performance, the Decision Tree’s high correlation coefficient proves its potential in mapping relationships within the dataset. Data augmentation, particularly using TVAE, was found to enhance the performance of the Random Forest model. The synthetic data generated by TVAE improved the original dataset, suggesting that data preprocessing and augmentation could be beneficial for improving model accuracy. The comprehensive search for hyperparameter tuning using Optuna, involving 300 trials for each model-data pair and each metric, ensured that our models were finetuned to their optimal configurations. This process showcased that it can influence model performance, as shown by the improved metrics across the models. For instance, the Random Forest model’s hyperparameters were tuned to balance depth and feature selection, which enhanced its performance on the augmented dataset. The results suggest that their effectiveness varied based on the differences in handling feature interactions, which indicates the potential need for a more refined feature engineering for hyperparameter tuning.
In Software Development Effort Estimation (SDEE), our approach significantly enhances predictive accuracy and addresses data variability challenges. We leverage advanced machine learning models like Random Forest Regressor, CatBoost Regressor, and Decision Tree Regressor, combined with data augmentation techniques such as Tabular Variational AutoEncoder (TVAE) and hyperparameter optimization using Optuna, to achieve notable improvements. Our methodology demonstrated superior reliability in effort predictions on the open-source agile TAWOS dataset, outperforming the Linear Regression baseline. The utilization of these sophisticated models ensures a robust handling of complex data patterns, leading to more accurate and reliable effort estimations. Furthermore, the integration of TVAE helps in overcoming data sparsity issues, providing a richer and more comprehensive dataset for training. Overall, our enhanced approach sets a new benchmark in SDEE, promoting efficiency and precision in project management and planning. The results of this study can significantly enhance software development project planning and management. Improved effort predictions allow for more accurate resource and deadline estimations, reducing delays and budget overruns. Using a Random Forest regressor on TVAE-augmented data provides high predictive accuracy, increases estimate reliability, and reduces the risk of resource assessment errors. These models help organizations better manage risks, optimize costs, and improve project efficiency. 6.1.
Despite these promising results, several limitations warrant attention. The computational complexity associated with TVAE augmentation and hyperparameter optimization is considerable, potentially restricting the scalability of our approach for larger datasets or real-time applications. Additionally, the skewed distribution of effort in the dataset, with most projects requiring relatively low effort and a few demanding significantly higher effort, poses challenges in achieving consistent predictive accuracy across all project types. 6.2.
Future research should focus on further enhancing SDEE models. This includes expanding the dataset through advanced feature engineering techniques to improve predictive power. Incorporating datasets from diverse sources and adopting different approaches such as Functional Point Analysis and Use Case Point Analysis can increase the models’ applicability across various project types. Moreover, adapting the models for real-time effort estimation is crucial to ensure accurate predictions in dynamic project environments. Addressing these limitations will enable future work to build on our findings and develop more effective and scalable SDEE solutions. For instance, exploring the use of different representations of graph neural networks or subsets of recurrent neural networks such as Fuzzy Cognitive Maps to perform WHAT-IF simulations for various scenarios could significantly enhance the robustness and applicability of SDEE models [21]. Ultimately, this would contribute to more successful software project management and resource allocation.
Acknowledgement This work is partially supported by CERCIRAS “Connecting Education and Research Communities for an Innovative Resource Aware Society” COST Action CA19135 funded by COST Association. [17] Chakraborty, S., Krishna, R., Ding, Y., & Ray, B. Deep learning based vulnerability detection: Are we there yet?. IEEE Transactions on Software Engineering, 48(9), 3280-3296, 2021. [18] Tadapaneni, P., Nadella, N. C., Divyanjali, M., & Sangeetha, Y. Software defect prediction based on machine learning and deep learning. In 2022 International Conference on Inventive Computation Technologies (ICICT) (pp. 116-122). IEEE, July, 2022. [19] Wang, H., Zhuang, W., & Zhang, X. Software defect prediction based on gated hierarchical
LSTMs. IEEE Transactions on Reliability, 70(2), 711-727, 2021. [20] Rimal, Y., Sharma, N., & Alsadoon, A. The accuracy of machine learning models relies on hyperparameter tuning: student result classification using random forest, randomized search, grid search, bayesian, genetic, and optuna algorithms. Multimedia Tools and Applications, 1-16, 2024. [21] Maden, A., & Yücenur, G. N. Evaluation of sustainable metaverse characteristics using scenariobased fuzzy cognitive map. Computers in Human Behavior, 152, 108090, 2024.