Augmenting Window Contents with Transfer Learning for Efort Estimation

Abstract

BACKGROUND: Some studies showed filtering out old completed projects with a window was efective for preparing a training dataset of an efort estimation model. Other studies showed selecting completed projects similar to a target project was also efective. The application of the similarity-based selection after the windowing approach was failed to synthesize their efects. The shortage of similar projects in the windowed pool was a potential cause of the failure. AIMS: To examine whether augmenting the window pool is efective to improve the estimation accuracy. METHOD: The moving windows approach was used for preparing a window pool. The similarity-based selection was applied to augment the pool. The selection assumes that projects in the pool form a set of virtual target projects. Old projects outside the pool were assumed to form a set of cross-company projects to be selected. The empirical study with a single-company ISBSG data was conducted to evaluate the efect. RESULTS: A positive synergistic efect was observed. The augmented window could synthesize the windowing approach and the similarity-based selection. It could also be combined with the similarity-based selection without performance degradation. CONCLUSIONS: Practitioners should consider adding projects similar to recently completed projects when efort estimation is based on historical data.

Keywords

• eol>efort estimation
• moving windows
• augmenting windows

1. Introduction

RQ1: Does augmenting moving windows with a relevancy filtering afects the estimation accuracy? RQ2: Does using the augmentation as a chronological filtering afect the estimation accuracy of the past combination method? most recently completed

Methodology

2. Related

Work

2.1. Chronological Filtering

Although research in software efort estimation models has a long history, relatively few studies have taken into consideration the chronological order of projects. Therefore, chronological filtering has not been studied well compared with other topics in efort estimation.

To our knowledge, Kitchenham et al. [ 4 ] were first to suggest the use of chronological filtering. They built four linear regression models with four subsets, each of which comprised projects from diferent ranges of time duration. As the coeficients of the models were diferent from each other, they allowed to drop out older project data. Lokan and Mendes [ 2 ] were the first to study the efect of using moving windows in detail. They used linear regression (LR) models and a singlecompany dataset from the ISBSG repository. Training sets were defined to be the projects. They found that the use of a window could afect accuracy significantly; predictive accuracy was better with larger windows; some window sizes were particularly efective. Amasaki and Lokan also investigated the efect of using moving windows with Estimation by Analogy [ 5 ] and CART [ 6 ]. They found that moving windows could improve the estimation accuracy, but the efect was diferent than with LR.

Recent studies showed the efect and its extent could be afected by windowing policies [ 7 ] and software organizations [ 8 ]. Lokan and Mendes [ 7 ] investigated the efect on accuracy when using moving windows of various ranges of time duration to form training sets on which to base efort estimates. They also showed that the use of windows based on duration could afect the accuracy of estimates, but to a lesser extent than windows based on a fixed number of projects [ 8 ].

2.2. Relevancy Filtering

Relevancy filtering is a type of transfer learning approach. While many filtering approaches have been a few studies on cross-company efort estimation have evaluated the efects of relevancy filtering approaches.

Turhan and Mendes [ 3 ] applied brings a so-called NN-filter [ 10 ] to cross-company efort estimation of web projects. They showed that an estimation model based on raw cross-company data was worse than that based on within-company data but was improved as comparable one by using the NN-filter. Kocaguneli et al. [ 11, 12, 13 ] also introduced a transfer learning approach called TEAK for improving cross-company effort estimation. They applied it to transfer old project to a new project and found that TEAK was efective not only for cross-company efort estimation but also for cross-time efort estimation [ 14 ].

NN-filter is based on a nearest neighbor algorithm. In that sense, a study by Amasaki and Lokan [ 5 ] can be considered an evaluation study of the combination of the relevancy filtering and the chronological filtering. In that study, the combination worked well to improve estimation accuracy for a narrow range of window sizes. While that study used a wrapper approach for feature selection and logarithmic transformation in addition to the nearest neighbor algorithm, our study aims to explore the efects of the combination without such complicated factors. For that purpose, we adopted two simple estimation techniques that were not adopted in [ 5 ], described in the next section.

3.1. Efort Estimation Techniques

In [ 1 ], average and median were used as software effort estimation models. The average was adopted because it uses the whole training set and is sensitive to the distribution of efort values in the training set. The median was adopted because it is robust to the distribution and contrasts with the average. These models estimate eforts without adjustments based on feature variables of projects.

To examine the diference in the use of feature variables in software efort estimation, we also adopted Lasso [ 15 ] for our experiment. Lasso is a kind of penalized linear regression models. Past studies on the chronological filtering used Lasso and showed that the chronological filtering was efective with it. Our experiment used LassoLarsIC of scikit-learn library.

3.2. Chronological Filtering

This study adopted fixed-size moving windows [ 2 ] and as past studies. ished projects were selected as a training set by the ifxed-size moving windows. The fixed-duration moving windows selected the latest projects finished within moving windows, we explored various values as well influences on the efectiveness of proposed for cross-project defect prediction (e.g., [ 9 ]), fixed-duration moving windows [ 8 ]. The latest ifnThe procedure of NN-filter is as follows:

3.5. Combination

1. Select closest instances of history data to each The combination of the chronological filtering and the instance of target project data in terms of un- relevancy filtering was investigated in [ 1 ]. The chronoweighted Euclidean distance. logical filtering and the relevancy filtering were com2. Combine the selected instances without dupli- bined as follows:

cation.

Note that each feature of project data was normalized with min-max normalization before the distance calculation.

As the synergistic efect could be observed with effective filtering, the relevancy filtering had to be conifgured as efective. For average and mean efort estimation models, we roughly fixed = 3, which is the smallest number which can make average and median estimations give distinct eforts. For lasso, we roughly ifxed = 10, half of the minimum of the window sizes we explored. In general, increasing would lead to worse estimation if NN-filter works well. Hence, these values could not be the best but were expected more reasonable than larger s. 1. Recently completed projects are selected with the moving windows approach. The remained old projects are discarded. 2. NN-filter is applied to select projects from the recently completed projects. The selected projects resemble a target project to be estimated. 3. The selected projects are used to train a software efort estimation model. 1. Recently completed projects are selected with As the chronological filtering relies on the time proxthe moving windows approach. imity, our experiment needs to assume a situation that 2. NN-filter is applied to select projects from the a development organization needs to respond to conremained old projects. The most similar project tinuously coming new projects. The size of windows to each project of the recently completed projects influences on where our experiment starts. As same as is selected. The set of selected projects has no the past studies, our experiment with a specific winduplicate. dow size was conducted as follows: 3. The selected projects and the recently completed

projects are combined. 4. The combined projects are used to train a software efort estimation model.

1. Sort all projects by starting date. 2. For a given window size , find the earliest project 0 for which at least + 1 projects were completed prior to the start of 0 (projects from 0 onwards are the ones whose training set is affected by using a window, so they form the set of evaluation projects for this window size. For example, with a window of 20 projects, at least 21 projects must have finished for the window to difer from the growing portfolio.) 3. For every project in chronological sequence, starting from 0, form a training set using moving windows and the growing portfolio (all completed projects).

• For no filtering, the training set is all projects that finished before started.

Note that NN-filter uses the efort variable in addition to feature variables. As eforts of the past projects are known, it is possible to use the efort variable in the augmentation process.

The augmentation shares the same assumption as the chronological filtering that the recently completed projects resemble a target project to be estimated. Results of NN-filter are also expected to pretend to be as fresh as the recently completed projects. Therefore, the selected projects are considered to keep the similarity to the target project.

Variable

Size Efort PDR

We concentrate first on the statistical significance of diferences in accuracy that arise from using the filtering approaches. To test for statistically significant differences between accuracy measures, we use the twoMin Mean Median Max StDev sided Wilcoxon signed-rank test (wilcoxon function 10 496 266 6294 699 of the scipy package for Python) and set the statisti62 4553 2408 57749 6212 cal significance level at = 0.05. The setting of this 0.53 16.47 8.75 387.10 31.42 study is a typical multiple testing, and the p-values of the tests must be controlled. Bonferroni correction is a popular method for this purpose. However, the adop• For fixed-size moving windows, the train- tion of this simple correction results in the lack of staing set is the most recent projects that tistical power, especially for not large efects. We thus ifnished before started. If multiple projects controlled the false discovery rate (FDR) of multiple ifnished on the same date, all of them are testing [ 18 ] with the “multipletests” function of included. the statsmodels package in Python. FDR is a ratio • For fixed-duration, the training set is the of the number of falsely rejected null hypotheses to most recent projects whose whole life cy- the number of rejected null hypotheses. cle had fallen within a window of months prior to the start of .

4. Results and Discussion

4. Estimate an efort of a target project based on past project data.

• For no filtering, the training set from the previous step is used.

4.1. Comparisons between Moving

Windows and Augmentation

5. Evaluate the estimation results.

This study used the single-company subset of the ISBSG dataset that was analyzed in [ 2, 7, 8, 5, 6, 16 ]. Table 1 shows summary statistics. We explored window sizes from 20 to 120 projects for the size-based moving windows and from 12 to 84 months for the durationbased moving windows as well as the past study [ 17 ]. = 0.05.

Figure. 1 revealed the efect of using the fixed-size No filtering, called the growing portfolio in past stud- moving windows and the augmentation, compared to ies, was used as a baseline for comparing the filtering always using the growing portfolio as follows: methods.

3.7. Performance Measures

The accuracy statistics that we used to evaluate the efort estimation models are based on the diference between estimated efort and actual efort. We used Mean Absolute Error (MAE), which is widely used to evaluate the accuracy of efort estimation models, as it is an unbiased measure that favours neither undernor over-estimates. • With average efort estimation, statistically significant diferences were found for almost all window sizes. The augmentation did not bring clear changes except for small window sizes, where additional statistically significant diferences were found. • With median efort estimation, no statistically significant diference was found for all window sizes. The augmentation improved the performance a bit for smaller window sizes but worsened it a bit for larger window sizes. The ef60 50 40 )(% 30 E eanA 20 isnm 10 ecn ffiree 0 D − 10 − 20 40 Window Size (number of projects) 100

60 80 (a) MW (average) 40 Window Size (number of projects) 100

60 80 (b) MW (median) 40 Window Size (number of projects) 100

60 80 (c) MW (lasso) 60 50 40 )(% 30 E eanA 20 isnm 10 ecn ffiree 0 D − 10 − 20 120 − 30 20 60 50 40 )(% 30 E eanA 20 isnm 10 ecn ffiree 0 D − 10 − 20 120 − 30 20 120 120 40 Window Size (number of projects) 100

60 80 (d) Augmentation(average) 40 Window Size (number of projects) 100

60 80 (e) Augmentation (median) 40 Window Size (number of projects) 100

60 80 (f) Augmentation (lasso) fects never caused a statistically significant difference. • With lasso, statistically significant diferences were found when window size is between 85 and 95 or is more than 110. The augmentation made the advantages in other window sizes statistically significant. The significant diferences in larger window sizes disappeared instead. Note that lasso was more accurate than the others even when used with the growing portfolio.

These observations suggested that the augmentation could bring a positive synergistic efect on the estimation accuracy when the augmentation was applied to ifxed-size windows with average or lasso.

Figure 2 plotted the same comparisons but using the ifxed-duration moving windows. In the figure, square points mean a statistically significant diference, with the fixed-duration moving windows being worse than the growing portfolio. These figures revealed the effects of the fixed-duration moving windows and the augmentation with it, compared to always using the growing portfolio as follows: than 40 months. The growing portfolio was no longer advantageous for larger window sizes. • With median efort estimation, the efective window range was more than 60 months. Disadvantageous window sizes are between 55 months and 60 months. The augmentation made the statistically significant diferences disappeared. • With lasso, there was no significant diference.

There was no clear advantage nor disadvantage.

The augmentation made no statistically significant diference while the diference got closer a bit.

These observations suggested that the augmentation could improve the estimation accuracy when the augmentation was applied to fixed-duration windows with average efort estimation.

The answer to RQ1 is yes: Augmenting moving windows with a relevancy filtering was useful. It did not cause an apparent negative synergistic efect, at least.

It sometimes made positive synergistic efects. • With average efort estimation, the efective window range was between 20 months and less than 30 months. The growing portfolio got advanta- The combination of the augmented moving windows geous for more than 53 months. The augmenta- and the NN-filter was evaluated under the same sitution extended the advantageous range to more ations. The number of neighbors was set to 3 for av

4.2. Evaluation of Combination of

Augmented MW and NN-filter

60 50 40 )(% 30 E eanA 20 isnm 10 cen ffiree 0 D − 10 − 20 − 30 60 50 40 )(% 30 E eanA 20 isnm 10 cen ffiree 0 D − 10 − 20 − 30 20 30 Windo4w0 Size (cal5e0ndar mont6h0s) 70 80 20 30 Windo4w0 Size (cal5e0ndar mont6h0s) 70 80 20 30 Windo4w0 Size (cal5e0ndar mont6h0s) 70 80 (a) MW (average) (b) MW (median) (c) MW (lasso) 20 30 Windo4w0 Size (cal5e0ndar mont6h0s) 70 80 20 30 Windo4w0 Size (cal5e0ndar mont6h0s) 70 80 20 30 Windo4w0 Size (cal5e0ndar mont6h0s) 70 80 (d) Augmentation(average) (e) Augmentation (median) (f) Augmentation (lasso) erage and median efort estimation models and 10 for mented moving windows with it, compared to always lasso because lasso models, as described in Section 3.3. using the growing portfolio as follows:

Figure 3 has 6 plots showing the diference in mean absolute error against fixed-size window sizes using • With average efort estimation, the NN-filter made the NN-filter and using the combination of the aug- statistically significant diferences for almost all mented windows and the NN-filter. These figures re- window sizes. Combining the augmented movvealed the efects of using the NN-filter and the aug- ing windows with the NN-filter made no clear change except for small window sizes, where the 30 20 )(% 10 EA ean 0 m ffiirsceeennD−− 1200 − 30 − 40 20 30 Windo4w0 Size (cal5e0ndar mont6h0s) 70 80 − 40

20 30 Windo4w0 Size (cal5e0ndar mont6h0s) 70 80 − 40 (a) NN (average) (b) NN (median) 20 30 Windo4w0 Size (cal5e0ndar mont6h0s) 70 80

(c) NN (lasso) diferences got smaller from about -30% to -20%.

The significance of the diferences was retained, though. • With median efort estimation, the NN-filter made no clear change while it caused positive efects as depicted by the line running below the zero line for a wide window range. Combining the augmented moving windows with the NN-filter made no clear change except for small window sizes. No statistically significant diference appeared. • With lasso, the NN-filter made no statistically significant change though it worsened the performance. Combining the augmented moving windows with the NN-filter mitigated the degradation. Note that the augmentation made the significant improvement as shown in Fig. 1(f).

The NN-filter canceled the improvement.

In [ 1 ], the combination of the moving windows and NN-filter caused a negative synergistic efect. For example, less than half of the window sizes could achieve the improvement of -30% or more where mean efort estimation was applied. The augmentation made the performance improvement of -30% or more for more than a half of the range as shown in Fig. 3(d). Therefore, these observations suggested that the augmented moving windows did not result in a negative synergistic efect caused by the combination of fixed-size moving windows and NN-filter.

Figure 4 plotted the same comparison but using the ifxed-duration moving windows. These figures revealed the efects of using the NN-filter and the augmented moving windows with it, compared to always using the growing portfolio as follows: • With average efort estimation, NN-filter made statistically significant diferences for almost all window sizes. Combining the augmented moving windows with the NN-filter made no clear change. • With median efort estimation, NN-filter made no clear change while it caused positive efects as depicted by the line running below the zero line for a wide window range. Combining the augmented moving windows with the NN-filter made no clear change. • With lasso, NN-filter made no statistically significant change though it worsened the performance. Combining the augmented moving windows with the NN-filter mitigated the degradation by NN-filter.

Therefore, these observations suggested that the augmentation did not result in a negative synergistic efect caused by the combination of fixed-size moving windows and NN-filter. Rather the degradation by NNiflter could be mitigated.

The answer to RQ2 is as follows: The combination of the augmented chronological filtering and the relevancy filter did not bring a negative synergistic efect except for small window sizes. Rather, the negative effect caused by the relevancy filtering was mitigated by the augmentation.

5. Conclusion

We explored the efects of the augmentation and its combination with a relevancy filtering for efort estimation. We confirmed the augmentation was a useful way to bring a positive synergistic efect of the chronological filtering and the relevancy filtering. Combining the augmented windows with the relevancy filtering, as well as in [ 1 ] also diminished the negative synergistic efect caused by the combination of the moving windows and NN-filter found in a past study. We thus concluded that the augmentation can be a good way to combine the two filtering approaches and also a good extension of the moving windows, which can be safely combined with the relevancy filtering.

Further investigation considering other transfer learning approaches is in future work. The NN-filter used for augmentation is a type of transfer learning, it is interesting to examine the efects of other approaches such as [ 14 ] for augmentation. Some transfer learning approaches for cross-project defect prediction [ 19 ] can also be applied. The threat to external validity can be mitigated with additional project data.

Acknowledgments

This work was partially supported by JSPS KAKENHI Grant #18K11246.

References

[1] S. Amasaki , Exploring Preference of Chronological and Relevancy Filtering in Efort Estimation , in: Proc. of Profes 2019 , Springer, 2019 , pp. 247 - 262 .

[2] C. Lokan , E. Mendes, Applying moving windows to software efort estimation , in: Proc. of ESEM 2009 , 2009 , pp. 111 - 122 .

[3] B. Turhan , E. Mendes , A Comparison of CrossVersus Single-Company Efort Prediction Models for Web Projects , in: Proc. of SEAA , IEEE, 2014 , pp. 285 - 292 .

[4] B. Kitchenham , S. Lawrence Pfleeger , B. McColl , S. Eagan , An empirical study of maintenance and development estimation accuracy , The Journal of Systems & Software 64 ( 2002 ) 57 - 77 .

[5] S. Amasaki , C. Lokan , The Efects of Moving Windows to Software Estimation: Comparative Study on Linear Regression and Estimation by Analogy , in: Proc. of IWSM-MENSURA 2012 , IEEE, 2012 , pp. 23 - 32 .

[6] S. Amasaki , C. Lokan , The Efect of Moving Windows on Software Efort Estimation: Comparative Study with CART , in: Proc. of IWESEP 2014 , IEEE, 2014 , pp. 1 - 6 .

[7] C. Lokan , E. Mendes, Investigating the Use of Duration-Based Moving Windows to Improve Software Efort Prediction , in: Proc. of APSEC 2012 , 2012 , pp. 818 - 827 .

[8] C. Lokan , E. Mendes, Investigating the use of duration-based moving windows to improve software efort prediction: A replicated study , Inf. Softw. Technol . 56 ( 2014 ) 1063 - 1075 .

[9] S. Herbold , CrossPare: A tool for benchmarking cross-project defect predictions , in: Proc. of 30th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) , IEEE, 2016 , pp. 90 - 95 .

[10] B. Turhan , T. Menzies , A. B. Bener , J. Di Stefano , On the relative value of cross-company and within-company data for defect prediction , Empirical Software Engineering 14 ( 2009 ) 540 - 578 .

[11] E. Kocaguneli , T. Menzies, How to Find Relevant Data for Efort Estimation? , in: Proc. of ESEM , IEEE, 2011 , pp. 255 - 264 .

[12] E. Kocaguneli , G. Gay, T. Menzies , Y. Yang , J. W. Keung , When to use data from other projects for efort estimation , in: Proc. of ASE , ACM, 2010 , pp. 321 - 324 .

[13] E. Kocaguneli , T. Menzies , A. B. Bener , J. W. Keung , Exploiting the Essential Assumptions of Analogy-Based Efort Estimation , IEEE Transactions on Software Engineering 38 ( 2012 ) 425 - 438 .

[14] E. Kocaguneli , T. Menzies , E. Mendes, Transfer learning in efort estimation , Empirical Software Engineering 20 ( 2015 ) 813 - 843 .

[15] R. Tibshirani , Regression shrinkage and selection via the lasso , J. Roy. Statist. Soc. Ser. B ( 1996 ) 267 - 288 .

[16] S. Amasaki , C. Lokan , Evaluation of Moving Window Policies with CART , in: Proc. of IWESEP 2016 , IEEE, 2016 , pp. 24 - 29 .

[17] S. Amasaki , C. Lokan , A Replication of Comparative Study of Moving Windows on Linear Regression and Estimation by Analogy , in: Proc. of PROMISE , ACM Press, 2015 , pp. 1 - 10 .

[18] Y. Benjamini , D. Yekutieli , The control of the false discovery rate in multiple testing under dependency , Annals of statistics 29 ( 2001 ) 1165 - 1188 .

[19] S. Herbold , Training data selection for crossproject defect prediction , in: Proc. of PROMISE '13 , ACM , 2013 , pp. 6 : 1 - 6 : 10 .