Analyzing the Performance of Two COSMIC Sizing Ap- proximation Techniques Using FUR at the Use Case Level Francisco Valdés-Souto National Autonomous University of Mexico Science Faculty CDMX, Mexico City, Mexico fvaldes@ciencias.unam.mx +52 55 56223899 ext 45735 ORCID: 0000-0001-6736-0666 Abstract. For accurate results, standards for the measurement of the functional size of software require that the functionality to be measured be fully known. However, when estimating in the early phases of software development where there is a lack of detail, approximate sizing techniques must be used. An approx- imation mechanism that has proven useful when there is no historical data is the technique of approximation by EPCU, there are two EPCU contexts with the range of the output variable other than 16.4 CFP and 44 CFP. Previous studies have shown that when functional requirements are at a granu- larity level of Functional Process, the context recommending being applied is that the output variable has a cut-off at 16.4 CFP, this is done when comparing the distribution of approximation results against the distribution of the REAL sizes. This paper investigates the two EPCU contexts defined in the literature, seeking to identify which technique appears to better represent the distribution of the REAL sizes when the granularity level was Use Cases (UC), the ‘Equal Size Bands’ (ESB) approximation and fuzzy logic-based approximation technique (EPCU) were also compared to identify which technique appears to represent the distribution of the REAL sizes better, when the granularity level was Use Cases (UC). From the results, it is not clear which approximation technique has the best per- formance, however carrying out the non-parametric test, it is possible to confirm statistically that the distribution of the EPCU44 approximation technique dis- plays behavior similar to that of the distribution of the COSMIC REAL sizes. Keywords. COSMIC ISO 19761; Approximate Sizing; Functional Size; EPCU Model. 70 F. Valdés-Souto 1 Introduction Functional Size Measurement (FSM) methods work best when the information to be measured – the functional user requirements – is fully known. Santillo [1], for instance, indicates that the “functional size of software to be developed can be measured pre- cisely [only] after the functional specification stage: this stage is often completed rela- tively late in the development process.” However, when estimating in the early phases of software development projects, there is often a lack of detailed information, which hinders the rigorous application of the measurement rules prescribed in international standards [1, 2, 3]. As observed by Desharnais et al. [4], when software documentation is lacking, it is not possible to apply all of the detailed measurement rules as specified in the interna- tional standards for the measurement of the functional size of software. Thus, in such early phases of the development cycle, to tackle this lack of detail and determine a relevant range of candidate functional size, measurers must fall back on approximation techniques for sizing requirements. As Vogelezang points out [5], “a rapid size measurement will be acceptable if it can be produced faster and still can deliver a reliable approximation of the detailed size measurement.” Most currently available approximation techniques for sizing the functional size of software requiring calibration employ historical data for better results in local contexts, such as the Equal Size Bands (ESB) approach described in [11]. However, collecting such data may be both expensive and time-consuming [8], and approximation tech- niques based on historical data are of little use without such data. This situation fre- quently occurs in the software industry. Additionally, COSMIC size approximation techniques were initially developed with a small sample of Functional Process (FP)/Use Cases (UC). To tackle this situation, a different approximation approach using fuzzy logic, re- ferred to as the EPCU COSMIC size approximation technique was proposed by Valdés et al. [9, 10, 11]. This approach does not require local calibration and is useful when there are no historical data available. Additionally, it is less expensive than the calibra- tion of the ESB approach or any other approximation approach that requires historical data [8, 9, 10]. Research on the EPCU size approximation technique has focused on two granularity levels [11, 12] of the Functional User Requirements (FUR) description: Functional Pro- cess [7] and Use Case [12], with different EPCU context definitions, especially about changing the domain of its output variable function. In order to analyze which of both EPCU contexts utilized and previously docu- mented [8, 9, 10] exhibits, a better performance for each granularity level of the FUR description, in 2017 Valdés [13] investigated and compared using non-parametric test- ing, which of the EPCU contexts (with upper size boundaries at 16.44 CFP1 as defined in [9] and 44 CFP as defined in [10]) appears to better represent the distribution of the REAL sizes, when the granularity level description was Functional Process. 1 In this paper when functional size unit is CFP, the version is v4.0.1. 71 IWSM/Mensura’18, September 18–20, 2018, Beijing, China This paper presents a case study with a more extensive set of Use Cases aiming to identify which of the approximation techniques (the ESB technique and the EPCU tech- nique using two distinct upper size boundaries) perform best, which means, statistically demonstrating which values distribution from the approximation techniques is more similar to REAL functional size distribution employing the standard COSMIC method, when the functional requirements are at the granularity level of Use Cases, a situation that presents very often in the industry. It is known that there is no standard definition for Use Case; however, it has been observed that frequently, Use Cases correspond to more than one Functional Process, considering the results in [13], where the EPCU context with upper size boundaries at 16.4 CFP (EPCU16.4) appears to represent the distribution of the REAL sizes better, and when the granularity level description was Functional Process, the hypothesis for this work was the following: H: The EPCU context with upper size cut-off at 44 CFP (EPCU44) better represents the distribution of the REAL sizes, when the granularity level of the functional user requirements description was Use Cases. The structure of this paper is organized as follows. Section II presents related work. Section III presents the experiment. Section IV presents the data including statistical analysis, while Section V, the conclusions with suggestions for further work. 2 Related work on functional size approximation techniques 2.1 Approximation techniques based on averages The IFPUG Function Point Analysis approximation technique for sizing was initially proposed in 1992 by Bock [16]. In 1997, Meli [14] proposed two variants but did not report on their performance. In 2003, Desharnais et al. [4] analyzed two approximation techniques commonly used in the industry: Function Points Simplified (FPS) [15], and Backfiring from lines of code [16]. Using the detailed data from this study (e.g., 90 business information projects from five organizations), the FPS technique, with average weights for each of the five function types of the IFPUG Function Points method, exhibited better perfor- mance (MMRE = 10.4%2 and PRED (0.15) = 76.2), while results from the Backfiring approach were highly inconsistent. In 2004, Conte et al. [3] extended the Early & Quick (E&Q) technique to the COSMIC FSM method and indicated that further tests would be needed to make ad- justments to the proposal, or to confirm it. This E&Q technique is based on (direct) analogy and (derived) analysis. It is a human-based size approximation technique im- pacted by the ability to “recognize” which components of the system belong to the proposed classes [17]. Since 2007, in the COSMIC document “Related Topics” [18] that evolved in 2015 into the COSMIC Guideline for Early or Rapid COSMIC Functional Size Measurement 2 MMRE, PRED(0.15) calculations using the detailed data from [4] 72 F. Valdés-Souto [6], two approximation techniques were based on averages where documented: the av- erage Functional Process approach, and the average Use Case approach. 2.2 Approximation techniques based on size bands In 2007, in a study of 50 projects, Vogelezang et al. [5] reported on a proposed size approximation technique based on size bands using the quartile approach. The authors also investigated the influence of distinct factors in approximate sizing and reported that, within this sample, the sole factor that exerted a substantial influence on the size of an average Functional Process in each of the quartiles was the number of Functional Processes [5]. In their case study, a reference software system with a full set of stable requirements and stated measured functional size was available. In general, an approach to approximate the size of a scaling factor for FUR type(s) of artifact(s) must be defined locally [18]. This requires, for instance, that an average size of the artifacts to be measured be established locally. This scaling factor represents the size that one can expect to be measured when FUR are at a level of detail where an accurate measurement can be made because all neces- sary details are available [5]. This solution requires historical data to produce an ade- quate scaling factor. In 2011, Santillo [1] proposed the Early and Quick COSMIC sizing approximation, based on earlier work [3] and the Analytic Hierarchy Process [19], a technique, which provides a means for making choices among sizing alternatives. In 2013, Almakadmeh [17] designed a framework to assign scaling factors for iden- tifying the granularity level of documentation of the functional requirements. Two var- iants of criteria for assessing granularity levels were defined: the first considered a functional component of software, and the second, the elements of a UML use-case model. To rank the levels of granularity identified, the scaling factors used in [5] were selected. Next, scaling factor assignment was based on conducting an analogy-based comparison with similar pieces of software in which the functional size of the software pieces was accurately measured using the COSMIC measurement method. In 2014, De Vito et al. [20] proposed a simplified measurement process (Quick/Early) that addressed the need for a simplified and rapid COSMIC measurement avoiding the use of scaling factors, where incorrect calibrations of scaling factors can lead to inaccurate approximations. The Quick/Early approximation approach can be applied on Use Case models to reduce measurement time. Quick/Early precision is di- rectly proportional to the granularity level of the Use Case model analyzed. This means that Use Cases require stable requirements that, however, do not occur too frequently in the early stages. Nonetheless, the authors concluded that Quick/Early accuracy is adequate. 2.3 Approximation techniques base on fuzzy logic In 2012, Valdés et al. [9] proposed a COSMIC size approximation solution using a fuzzy logic model referred to as the Estimation of Projects in a Context of Uncertainty (EPCU) [2, 21, 22]. 73 IWSM/Mensura’18, September 18–20, 2018, Beijing, China The advantages of the EPCU size approximation technique can be summarized as follows [ 8, 9, 10]: ─ Does not require local calibration and is useful when there are no historical data available. ─ Less expensive to calibrate than the ESB approach, which requires historical data. ─ Exhibits good behavior, even when individuals are not acquainted with the COSMIC method. ─ Exhibits good behavior, even when requirements are not fully known. ─ Enables systematic replication of the information. In these studies [2, 21, 22], two EPCU contexts were defined for a continuous range of possible values with a “natural” upper boundary, or cut-off instead of size bands, and a mixture of granularity levels (Functional Process and Use Case), simulating the early phases of the software life cycle: 1. The first EPCU context, defined a cut-off at 16.4 CFP [8, 9] (EPCU16.4), based on the ESB approach as defined by Vogelezang [5] (Small = 4.8 CFP, Medium =7.7 CFP, Large = 10.7 CFP, and Very Large = 16.4 CFP), and 2. The second context defined a cut-off at 44 CFP [11] (EPCU44), defined after ana- lyzing the database used by Vogelezang [5], that contains two general analyses over the functional process measured labeled Q-Size and Q-Number. Considering the Q- Size where the total measured size is divided into quartiles and the average FP size is calculated from each one (Q1=3.7 CFP, Q2=7.7 CFP, Q3=14.6 CFP and Q4=44.1 CFP) For this new study, it is considered the integrated analysis, the concept of both is described below. EPCU approach research also focused on the definition of the EPCU context, select- ing several samples from case studies, usually an industry or reference project with fewer than 12 practitioners, focusing on analyzing the performance of the approxima- tion technique in the early phases. For instance, Valdés et al. [10] reported on a case study of a simulation of early approximation using the EPCU model for an industry project for which only the names of the Use Cases were made available to participants. This case study confirmed that the EPCU size approximation approach does not require local calibration and is useful when there are no historical data available. Besides, it proved less expensive than cali- bration of the ESB approach, which requires historical data. In this case study, the out- put variable was defined for a continuous range of possible values with an upper bound- ary, or cut-off instead of size bands, at 16.4 CFP, as per the ESB approach defined by Vogelezang et al. [5]. For a case study with a REAL industrial project, the EPCU size approximation technique yielded better results than the ESB approach, while both tech- niques led to lower sizes than the real functional size. In 2015, Valdés et al. [11] proposed another version of their fuzzy logic size approx- imation technique. It defined a continuous range of possible values for the output vari- able with an upper Q4 (4th Quartile) cut-off of 44 CFP for a Functional Process using 74 F. Valdés-Souto the dataset of Vogelezang et al. [5]. For the study of an industry project that considered Use Case granularity level, the EPCU cut-off at 44 CFP [11] yielded better results on comparison with the ESB approach and EPCU cut-off at 16.4 CFP [10]. The Functional size was underestimated for Functional Process or Use Cases using the EPCU cut-off at 16.4 CFP. On the other hand, results were above and below the REAL value for Use Cases using the EPCU cut-off at 44 CFP. More realistic results were obtained using the EPCU44. Research on the EPCU size approximation technique has focused on two granularity levels [11, 12] of the FUR description: Functional Process [7], and Use Case [12], using two EPCU context definitions; however, it was not clear when to utilize each EPCU context (EPCU16.4, EPCU44), in order to analyze which of the two has a better performance for each granularity level of functional requirements. In 2017, Valdés [13] investigated and compared using a non-parametric test, which of the EPCU contexts appeared to represent the distribution of the REAL sizes better, when the granularity level was Functional Process. In the study [13], it was statistically demonstrated that distribution for approximation values using EPCU16.4 was similar to REAL value distribution employing the standard COSMIC method with 180 Functional Process. There is no standard definition for Use Case, and it has been observed that frequently that Use Cases involve more than one Functional Process, sounds logical that the EPCU approximation technique with a cut-off of 44 CFP might be more useful if functional requirements are at the granularity level of Use Cases, a situation that occurs very fre- quently in the industry. However, based on the findings of [13], the valid conclusion is that the EPCU44 approach is not as useful with the Functional Process level of gran- ularity, as it leads to oversizing, and a similar assessment, but employing Use Cases, is proposed as further work. 2.4 Smmary of COSMIC approximation techniques The validity of the majority of approximation techniques is dependent on the represent- ativeness of the samples with respect to the software being approximated. In other words, the majority of approximation methods require local calibration, and this re- quires local historical data. Even more COSMIC size approximation techniques were initially developed with a small sample of data. However, as pointed out by Morgensht- ern [8]: “Algorithmic models need historical data, and many organizations do not have this information. Additionally, collecting such data may be both expensive and time- consuming.” Approximation techniques based on historical data are of little use for organizations without such data. Alternatives must, therefore, be developed for such contexts of approximation. The COSMIC Guideline for Early or Rapid COSMIC Functional Size Measurement [6] integrates several techniques for the approximate sizing of new, ‘whole’ sets of re- quirements. The approximation techniques described in [6] include approximation techniques based on size bands or based on average. 75 IWSM/Mensura’18, September 18–20, 2018, Beijing, China The majority of the techniques presented in [6] are based on the existence of histor- ical data to determine the scaling factor (average, or size bands) or another calibration, and that there are stable requirements [11]. 2.5 Impact of approximated size on the estimation of effort In 2013, De Marco et al. [23] investigated to what extent some COSMIC-based approx- imate sizing could be useful for project managers for early effort estimation for Web applications. The authors reported an empirical analysis employing data from 25 Web applications to assess whether two approximate sizes (number of COSMIC Functional Processes (FP) or the Average Functional Process approach) could be exploited to ac- quire accurate effort estimates. These authors concluded that COSMIC-based approxi- mate sizing was a suitable approach for early effort estimates, while estimates obtained with approximate sizes were worse than those achieved employing the size obtained from the application of the standard COSMIC method. 3 Experiment with approximation techniques This section describes the experiment carried out to evaluate the size approximation techniques and identify which technique appears to represent the distribution of the REAL sizes better, when the granularity level was Use Cases (UC). 3.1 Context and participants As a part of a consultancy project whose objective was to implement the use of COSMIC for a Government entity in Mexico carried out in 2016, with the objective of generating formal estimation models, several projects were measured using the COSMIC method. The three main circumstances described in [6], in which only an approximate COSMIC functional size may be possible were presented in the project: ─ When a size measurement is needed rapidly, and an approximate size measurement is acceptable if it can be measured much faster than with the standard method. This is known as ‘rapid sizing’; ─ Early in the life of a project before the actual requirements have been specified in enough detail for precise size measurement. This is known as ‘early sizing’; ─ In general, when the quality of the documentation of the actual requirements is not sufficiently good for precise size measurement. Considering the information below, the functional size for the projects was gathered using the approximation approaches as the first step and then, when the required detail for the requirements was accomplished, the full standard was used to obtain the func- tional size. 76 F. Valdés-Souto To conduct a comparison with the previous study [13] focused on 180 Functional Process, four projects were selected. These four projects integrated 293 Use Cases that were approximated using ESB and EPCU techniques. The people in the Government entity received 24 hours of training in COSMIC dur- ing the consultancy project, including the EPCU approximation technique and that of equal size bands. The information required for using the approximation techniques were required from the technical people, specifically from the project leader for each project, with a distinct project leader for each project. It is important to mention that the techniques related to the Requirements Engineer- ing used by the Government entity was not affected by the consultancy and was possible to observe that sometimes the Use Cases include much functionality. Table 1 shows the number of Use Cases by project. Table 1. Use Cases by project considered in the case study ID Project # UC Assigned 1 43 2 96 3 55 4 99 Total 293 3.2 Participant instructions for functional size measurement and approximation Each project leader was asked to perform the following: 1. Identify, for each project, the set of Use Cases assigned to be developed. 2. Classify (using expert judgment) by size each of the Use Cases using the following linguistic values: Small; Medium; Large, and Very Large3. 3. Classify (using expert judgment) the number of objects of interest for each of the Use Cases using the following linguistic values: Few; Average, and Many. 4. Assign values (using expert judgment) in the range 0 - 5 ε R for the two previously classified input variables (points 2 and 3, the Use Cases’ size, the number of objects of interest related to the Use Cases) defined within the EPCU context, considering the subjective classification relative to the functional size of the Use Cases (e.g., Step 2), and the subjective classification for the number of objects of interest in each Use Case (e.g., Step 3). 3 The linguistic values were defined in concordance to the ESB Approach to enabled the compa- rison. 77 IWSM/Mensura’18, September 18–20, 2018, Beijing, China 5. Measure functional size using the COSMIC method and provide the size for each Use Case. 3.3 Data collected by participants Project leaders identified 293 Use Cases in four projects (Table 1), and the data pro- vided by the project leaders were the following (see Appendix I for details): ─ A value assigned within the range of 0 - 5 ε R for the size of each Use Case. ─ A value assigned within the range of 0 - 5 ε R for the objects of interest for each Use Case. ─ COSMIC size using the COSMIC method for each Use Case. The linguistic classification of the Use Cases and the linguistic classification of the objects of interest for each Use Cases (data from Steps 2 and 3) were not included in the table in the Appendix since the input for the EPCU approximation approach were the values assigned for each variable (data from Step 4). 3.4 Researcher steps Using the linguistic classification (Small, Medium, Large, and Very Large) assigned by the participants for the Use Cases the ESB technique was performed. Using the values (between 0 and 5) assigned by the participants for the two input variables of the fuzzy logic based EPCU approximation technique, CFP units were per- formed by the researcher using the EPCU approximation technique with distinct EPCU contexts (EPCU16.4 and EPCU44) defined in [8, 9] and [11]. The COSMIC size approximated with the data provided by the project leaders was verified using the COSMIC measurement principles and rules by two consultants with more than 7,000 CFP measurement experiences at the verification moment. COSMIC functional size and approximate size for each Use Case are presented in Appendix II where: ─ Column 1 presents the Project identifier. For confidential purposes, the Projects were labeled sequentially, from “Proj 1” to “Proj 4. ─ Column 2 presents the Use Case identifier. For confidential purposes, the Use Cases were labeled sequentially, from “UC 1” to “UC 293. ─ Column 3 presents the functional size obtained utilizing the standard COSMIC method – in CFP units, ─ Column 4 presents the Equal Size Band approximation approach, ─ Column 5 presents the EPCU size approximation approach using an output variable domain function from 2 - 16.4 CFP [8] [9], and ─ Column 6 presents the EPCU size approximation approach using an output variable domain function from 2 - 44 CFP [10]. 78 F. Valdés-Souto 4 Data Analysis 4.1 Quality Criteria Three most frequently quoted quality criteria [24] were used to analyze the behavior of the two approximation techniques : ─ Mean Magnitude of Relative Error (MMRE), ─ Standard Deviation of MRE (SDMRE), and ─ Prediction level, here PRED(25%) was selected. The Median Magnitude of Relative Error (MdMRE) is also used. The primary ad- vantage of the median over the mean is that the median is not sensitive to the outliers. Table 2 presents the results for each of these quality criteria for each approximation approach (top line) for the set of 293 Use Cases: 1. With an MMRE of 61.4%, the ESB presented the best results (in comparison to MMRE = 65.7% with the EPCU16.4 technique and MMRE = 117.4% with EPCU44). 2. With an SDMRE of 49.1%, ESB presents the best results, in comparison to SDMRE of 62.2% for the EPCU16.4 technique and SDMRE = 156.1% for the EPCU44 tech- nique. 3. Within a PRED (25%) at 20.8%, the EPCU with a cut-off at 44 CFP presents the best results, in comparison to 18.8% with ESB and 17.1% with EPCU with the cut- off at 16.4 CFP. 4. With a MdMRE of 56.9%, the EPCU16.4 technique presents the best results, in com- parison to 59.5% with ESB and 63.3% with EPCU with a cut-off of 44 CFP. It is possible to observe that the difference between the maximal and the minimal MdMRE values are less than the other quality criteria. Two quality criteria present the best results in the ESB approach (MMRE and SDMRE); however, the prediction level presents the best results for the EPCU with the cut-off at 44 CFP, and the MdMRE presents the best results for the EPCU16.4. Table 2. Approximation technique performance for 293 Use Cases ESB EPCU 16.4 EPCU 44 MMRE 61.4% 65.7% 117.4% MdMRE 59.5% 56.9% 63.3% SDMRE 49.1% 62.2% 156.1% PRED(25%) 18.8% 17.1% 20.8% From the quality criteria, it is not clear which approximation technique has the best performance, this because the central tendency measurements are affected by outliers. 79 IWSM/Mensura’18, September 18–20, 2018, Beijing, China MMRE has been shown to be a biased estimator of central tendency of the residuals of a prediction system because it is an asymmetric measure [25], [26], [27], [28]. Shep- perd et al. [29] proposed the Mean Absolute Residual (MAR), which, unlike MMRE, is not biased to compare the accuracy of a given estimation method P against the accu- racy of a reference estimation method P0. 1 MAR = ∑𝑛𝑖=1 |𝑦𝑖 − 𝑦̂𝑖| (1) 𝑛 Based on the calculated MARP (the MAR of the proposed method) and MARP0 (the MAR of a reference method), Shepperd et al. [29] propose to compute a Standardized Accuracy measure (SA) for estimation method P. 𝑀𝐴𝑅𝑃 SA = 1 − (2) 𝑀𝐴𝑅𝑃0 Where values of SA close to 1 indicate that P outperforms P0, values close to zero indicate that P’s accuracy is similar to P0’s accuracy, and the negative values indicate that P is worse than P0. The authors [29] suggest to use a referenced model random based considering the known (actual) values of previously measured projects, however, Lavazza [30] observed that the comparison with random estimation is not very effective in supporting the evidence that P is a good estimation model. Instead, proposed to use a “Constant Model” (CM), where the estimate of the size of the ith project is given by the average of the sizes of the other projects, then the calculation of the MARCM of these estimates is realized, and then the compute of SA, comparing method P with a method CM, generalizing that SA could be used to compare an estimation method P against any other method P1 used as a reference method. 𝑀𝐴𝑅𝑃 SA = 1 − (3) 𝑀𝐴𝑅𝑃1 Table 3. Standardized Accuracy measure using the ESB as a reference (P1) EPCU 16.4 EPCU 44 MAR Calculated using (1), 16.7 17.6 The MAR for ESB = 17.7 SA Calculated using (3) -0.96 -1.05 Considering ESB as P1 Table 3 presents the results related to the comparison between each EPCU context (top line), considering the Standardized Accuracy measure approach proposed for Lavazza [30], using the ESB approximation approach as CM as in (3). With a SA close to zero (0.05 for EPCU 16,4 and 0.01 for EPCU 44), both EPCU context present similar accuracy to the reference approximation approach (ESB). Considering the SA measure, 80 F. Valdés-Souto the ESB present a better result, it is not clear which approximation technique has the best performance. 4.2 Graphical Analysis Fig. 1 A), graphically presents, for each of the 293 Use Cases, the REAL COSMIC size (in blue) and the size approximated with the ESB technique (in orange). Real v.s. ESB 400 350 300 250 200 150 100 50 0 0 50 100 150 200 250 300 Real ESB Real v.s. ESB 100 80 60 40 20 0 0 50 100 150 200 250 300 Real ESB Fig. 1. REAL COSMIC size vs. approximated size using the ESB technique – 293 Use Cases. A) Vertical axis boundaries at 400 CFP. B) Vertical axis boundaries at 100 CFP. 81 IWSM/Mensura’18, September 18–20, 2018, Beijing, China Note that with the ESB technique, the only four values possible for the approximated size (in orange) are as follows: 4.8 CFP; 7.7 CFP; 10.7 CFP, and 16.4 CFP correspond- ing to the four average size bands of Functional Process (Small, Average, Large, and Very Large). From the data (Appendix II, column 3), it is possible to conclude that 230 Use Cases (78.5%) were underestimated; in consequence, overestimated 63 Uses Cases are (21.5%). From these overestimated Use Cases, 139 are due to that the upper boundary, or cut-off, was established at 16.4 CFP and the Use Cases had a functional size higher than that of the cut-off. Fig. 2 depict the graphical comparison with the EPCU16.4 technique. This technique defines a continuous range of possible values between 2 CFP and an upper boundary or cut-off at 16.4 CFP; consequently, at least 139 Use Cases were underestimated be- cause of the upper boundary. Looking at the data from (Appendix II, column 3), overestimated Uses Cases num- bered 99 (33.8%), while underestimated Use Cases numbered194 (66.2%). It is possible to observe that the number of Use Cases underestimated decrease in 36 Use Cases con- sidering the ESB technique, and the Use Cases overestimated increase. Fig. 3 presents the graphical comparison with the EPCU44 technique because this approach has a cut-off at 44 CFP; naturally, fewer Use Cases were underestimated, 130 (44.4%), while overestimated Use Cases numbered 163 (55.6%), and for the EPCU44 technique, more Uses Cases were overestimated. Intuitively from the previous figures, the EPCU44 better represents the distribution of the REAL sizes; however, it is not easy to infer from Fig.1 to Fig. 3, because there are several outliers. This confirms the reason regarding the big difference between the maximal and the minimal values for MdMRE and MMRE from Table 2. Considering the difference between MdMRE and MMRE, it is possible to assume that the distribution is skewed and that the most representative value is the MdMRE, because central tendency measurements were affected by the outliers. In Fig. 4, the boxplots related to the REAL Value of functional size, and ESB, EPCU16.4, and EPCU44 functional size approximation, are presented. This is a better approach for analyzing the data without considering the outliers. From Fig. 4, it might be easier to infer that EPCU44 better represent the distribution of the REAL sizes, because both boxplots are very similar. 82 F. Valdés-Souto Real v.s. EPCU16.44 400 350 300 250 200 150 100 50 0 0 50 100 150 200 250 300 Real EPCU16.4 Real v.s. EPCU16.44 100 80 60 40 20 0 0 50 100 150 200 250 300 Real EPCU16.4 Fig. 2. REAL COSMIC size vs. approximated size using the EPCU-16.4 technique – 293 Use Cases. A) Vertical axis boundaries at 400 CFP. B) Vertical axis boundaries at 100 CFP. 83 IWSM/Mensura’18, September 18–20, 2018, Beijing, China Real v.s. EPCU44 400 350 300 250 200 150 100 50 0 0 50 100 150 200 250 300 Real EPCU44 Real v.s. EPCU44 100 80 60 40 20 0 0 50 100 150 200 250 300 Real EPCU44 Fig. 3. REAL COSMIC size vs. approximated size using the EPCU-44 technique – 293 Use Cases. A) Vertical axis boundaries at 400 CFP. B) Vertical axis boundaries at 100 CFP 4.3 Non-parametric test Considering the quality criteria affected by the central tendency measurements, the ap- proximation technique that provides better results was ESB. From the plots in Fig.s 1, 2, 3, and 4, the EPCU44 technique appears to better represent the distribution of the REAL sizes; however, this needs to be confirmed by statistical analysis. In non-parametric statistics, a well-known procedure for testing the differences among more than two related samples is the Friedman test [24, 25] The objective of the test is to determine whether it can be concluded, from a sample of results, that there is a difference among treatment effects [32]. 84 F. Valdés-Souto Using the Friedman non-parametric test to analyze whether there is a difference among the performances of distinct treatments across the same datasets for functional size, that is, whether the data distributions are equal, a null hypothesis H0 was defined as: H0: There are NO meaningful differences in the distributions of REAL, ESB, EPCU16.4, and EPCU44 datasets. In consequence, the alternative hypothesis was defined as: H1: At least one distribution (REAL, ESB, EPCU16.4, and EPCU44) is significantly different. A significance level of ɑ (alpha (ɑ)) = 0.05 was assumed. SPSS® version 22 software in the Spanish language was utilized to evaluate the Friedman test for the four distinct treatments (REAL, ESB, EPCU16.4, and EPCU44), and the results are summarized in Table 4. The full results from SPSS ® are presented in Appendix III. In Table 4, “N” represents the 293 Use Cases, “df” represents the degrees of freedom (with four distinct treatments; the df is 3 (#treatments -1)). Here, the statistical signifi- cance (“Asymp. Sig.” or p-value) is a very small number at E-101, thus below the re- quired significance level of ɑ =0.05. Therefore, the null hypothesis (e.g., H0: There are NO meaningful differences in the distribution of REAL, ESB, EPCU 16.4, and EPCU 44) is rejected, and it is possible to state that at least one treatment has a distinct distribution. Fig. 4. Boxplots related to the REAL Value of functional size, and ESB, EPCU16.4, and EPCU44 functional size Approximation. 85 IWSM/Mensura’18, September 18–20, 2018, Beijing, China Table 4. Friedman test results on the testing of the four distributions (REAL, ESB, EPCU16.4, and EPCU44) N 293 Chi-Square 468.936 df 3 Asymp. Sig. 2.5722E-101 In order to identify where the difference is, a post-hoc test is needed. In this instance, a post-hoc test assesses the difference between treatments as follows: ─ REAL and ESB ─ REAL and EPCU16.4 ─ REAL and EPCU44 ─ ESB and EPCU16.4 ─ ESB and EPCU44 ─ EPCU-16.4 and EPCU44 Here, the post-hoc test compared two treatments at a time. The Wilcoxon [32] test was executed using SPSS® software, and the Bonferroni correction [33] was consid- ered; thus, the ɑ value (ɑ =0.05) was divided by 4 because four distinct treatments were used. This means that the ɑ was reset at ɑ =0.0125. Considering the latter, the null hypothesis H0 for the post-hoc test was: H0: There are NO meaningful differences between the distributions for the two treat- ments compared (see the previous list). In consequence, the alternative hypothesis was defined as: H1: The distribution for the two treatments compared is significantly different, as- suming a significance level of ɑ = 0. 0125. Table 5 presents the results of applying the Wilcoxon test for two treatments in SPSS®. Column 1 indicates the comparison, and column 2, the significance for the Wilcoxon test. The significance value was compared with ɑ = 0.0125 by accepting (>ɑ = 0.0125) or rejecting (<ɑ = 0.0125) the null hypothesis; the results are presented in column 3. The full results from SPSS® are presented in Appendix IV. From Table 5, with a p-value of ɑ =0.0125, it is possible to confirm statistically that only the distribution of the EPCU44 approximation technique (with a cut-off at 44 CFP) displays a behavior similar to the distribution of the COSMIC REAL sizes (REAL value), considering the granularity level of Use Cases, which graphically could be ob- served in Fig. 4. 86 F. Valdés-Souto Table 5. Wilcoxon post hoc test results Comparison Asymp. Sig. Statistical Signif- icance (p-value) >ɑ =0.0125 REAL and ESB 4.8089E-30 NO REAL and 3.6339E-19 NO EPCU16.4 REAL and 0.281 YES EPCU44 ESB and 1.3091E-38 NO EPCU16.4 ESB and 8.7473E-50 NO EPCU44 EPCU16.4 and 8.2436E-50 NO EPCU44 5 CONCLUSIONS In this paper, using a large sample of 293 Use Cases from real projects, two approxi- mation techniques were evaluated to identify which performs best with this dataset larger, which is larger than previous sets mentioned in related works. This implies sta- tistically demonstrating which value distribution from the approximation techniques is more similar to REAL functional size distribution employing the standard COSMIC method, when the functional requirements are at the granularity level of Use Cases, a situation encountered very frequently in the industry. From the previous work [13], the EPCU context appears to represent the distribution of the REAL sizes better; when the granularity level was Functional Process, 180 Func- tional Process were used. From our findings related to quality criteria, it is not clear which approximation tech- nique executes the best performance, this is because the central tendency measurements are affected by outliers, and the sample has several outliers, as in reality occurs. It is well known that there is no standard definition for Use Case, and this could be a reason for the outliers. For instance, there are Use Cases with more than 100 or 300 CFP. The presence of outliers can be observed in Fig.s 1 - 4, even though, intuitively from the previous figures, the EPCU44 better represents the distribution of the REAL sizes. However, it is not easy to infer. 87 IWSM/Mensura’18, September 18–20, 2018, Beijing, China On carrying out the non-parametric test, it is possible to confirm statistically that only the distribution of the EPCU44 approximation technique displays behavior similar to that of the distribution of the COSMIC REAL sizes (REAL value), considering the granularity level of Use Cases, accepted the following hypothesis: H: The EPCU context with an upper size cut-off at 44 CFP (EPCU44) better repre- sents the distribution of the REAL sizes, when the granularity level of the FUR descrip- tion was Use Cases. Considering the findings and the previous work, it is possible to define when the granularity level of the FUR description was Use Cases, with our recommending the EPCU44 approximation approach, while when the granularity level of the functional user requirements description was Functional Process, the EPCU16.4 approximation approach is recommended. The research developed in this paper only includes two of the approximation tech- niques mentioned in the Guideline for Early or Rapid COSMIC Functional Size Meas- urement [6]; others should be investigated as well, using similar experiments. Because the spread of the use of agile practices, a similar assessment to that of this paper but employing User Histories as the granularity level of the functional user re- quirements description should be conducted. References 1. L. Santillo, Early and Quick COSMIC FFP Overview, in: A.A. Reiner Dumke (Ed.), Cosm. Funct. Points Theory Adv. Pract., CRC Press, Boca Raton, FL, USA, 2011: pp. 176–191. 2. F. Valdés-Souto, Design of a Fuzzy Logic Software Estimation Process, École De Technologie Supérieure, Université Du Québec, 2011. 3. L. Conte, M., Iorio, T., Santillo, E&Q: An Early & Quick Approach to Functional Size Measurement Methods, in: Istituto di Ricerca Internazionale (Ed.), Softw. Meas. Eur. Forum SMEF 2004, Rome, Italy, 2004: p. 416. 4. J.M. Desharnais, A. Abran, Approximation Techniques for Measuring Function Points, Proc, in: 13th Inter. Work. Softw. Meas. (IWSM 2003), Springer, Montreal, Canada, 2003: pp. 270–286. 5. F. Vogelezang, T. Prins, Approximate size measurement with the COSMIC method: Factors of influence, in: Softw. Meas. Eur. Forum SMEF 2007, Rome, Italy, 2007: pp. 167–178. 6. Common Software Measurement International Consortium (COSMIC)., Guideline for Early or Rapid COSMIC Functional Size Measurement, 2015. 7. Common Software Measurement International Consortium (COSMIC), Measurement Manual v4.0.1, 2015. 8. O. Morgenshtern, T. Raz, D. Dvir, Factors Affecting Duration and 88 F. Valdés-Souto Effort Estimation Errors in Software Development Projects, Inf. Softw. Technol. 49 (2007) 827–837. doi:10.1016/j.infsof.2006.09.006. 9. F. Valdés-Souto, A. Abran, Case Study: COSMIC Approximate Sizing Approach Without Using Historical Data, in: Jt. Conf. 22nd Int. Work. Softw. Meas. 2012 Seventh Int. Conf. Softw. Process Prod. Meas., IEEE, Assisi, Italy, 2012: pp. 178–189. doi:10.1109/IWSM- MENSURA.2012.34. 10. F. Valdés-Souto, A. Abran, COSMIC Approximate Sizing Using a Fuzzy Logic Approach: A Quantitative Case Study with Industry Data, in: F. Vogelezang, M. Daneva (Eds.), 2014 Jt. Conf. Int. Work. Softw. Meas. Int. Conf. Softw. Process Prod. Meas., Conference Piblishing Services (CPS), Rotterdam (Netherlands), 2014: pp. 282–292. doi:10.1109/IWSM.Mensura.2014.44. 11. F. Valdés-Souto, A. Abran, Improving the COSMIC Approximate Sizing Using the Fuzzy Logic EPCU Model, in: A. Kobylinski, B. Czarnacka-Chrobot, J. Swierczek (Eds.), Jt. Conf. 25rd Int. Work. Softw. Meas. 10th Int. Conf. Softw. Process Prod. Meas. - IWSM- MENSURA 2015, Springer International Publishing, Cracow (Poland), 2015: pp. 192–208. doi:10.1007/978-3-540-71649-5. 12. M. Fowler, UML Distilled, A Brief Guide to the Standard Object Modeling Language, Third Edit, Addison Wesley, n.d. 13. F. Valdés-Souto, Analyzing the performance of two COSMIC approximation sizing techniques at the functional process level, Sci. Comput. Program. 135 (2017) 105–121. doi:10.1016/j.scico.2016.11.005. 14. R. Meli, Early Function Points: A New Estimation Method for Software Projects, in: ESCOM 1997, Berlin, Germany, 1997: pp. 1–10. 15. D.B. Bock, Robert Klepper, FP-S: A Simplified Function Point Counting Method, J. Syst. Softw. 18 (1992) 245–254. 16. C. Jones, Applied Software Measurement, Assuring Productivity and Quality, 2nd ed., McGraw Hill, New York, NY, 1997. 17. K. Almakadmeh, Development of a Scaling Factors Framework to Improve the Approximation of Software Functional Size with COSMIC - ISO19761, École de Technologie Supérieure, Université Du Québec, 2013. 18. Common Software Measurement International Consortium (COSMIC)., Advanced and Related Topics, 2007. 19. L. Santillo, Early FP Estimation and the Analytic Hierarchy Process, in: ESCOM-SCOPE 2000, Munich, April, Munich, Germany, 2000: pp. 1– 20. 20. G. De Vito, F. Ferrucci, Approximate COSMIC Size: The Quick/Early Method, in: Proc. - 40th Euromicro Conf. Ser. Softw. Eng. Adv. Appl. SEAA 2014, IEEE, Verona, Italy, 2014: pp. 69–76. 89 IWSM/Mensura’18, September 18–20, 2018, Beijing, China doi:10.1109/SEAA.2014.30. 21. F. Valdés-Souto, A. Abran, Industry Case Studies of Estimation Models Using Fuzzy Sets, in: Reiner Dumke (Ed.), Softw. Process Prod. Meas. Int. Conf. IWSM-Mensura 2007, UIB-Universitat de les Illes Baleares, Illes Baleares, Spain, 2007: pp. 87–101. 22. F. Valdés-Souto, A. Abran, Comparing the Estimation Performance of the EPCU Model with the Expert Judgment Estimation Approach Using Data from Industry, in: R. Lee (Ed.), Softw. Eng. Res. Manag. Appl. 2010, Springer-Verlag, Berlin, 2010: pp. 227–240. 23. L. De Marco, F. Ferrucci, C. Gravino, Approximate COSMIC Size to Early Estimate Web Application Development Effort, in: 39th Euromicro Conf. Ser. Softw. Eng. Adv. Appl. Approx., Conference Publishing Services (CPS), Santander, Spain, 2013: pp. 349–356. doi:10.1109/SEAA.2013.41. 24. M. Friedman, The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance, J. Am. Stat. Assoc. 32 (1937) 675– 701. 25. L. Lavazza, Accuracy Evaluation of Model-based COSMIC Functional Size Estimation, in: ICSEA 2017 Twelfth Int. Conf. Softw. Eng. Adv., 2017: pp. 67–72. 26. B.A. Kitchenham, L.M. Pickard, S.G. MacDonell, M.J. Shepperd, What accuracy statistics really measure, in: IEE Proc. - Softw., 2001: pp. 81– 85. doi:10.1049/ip-rsn:20010506. 27. T. Foss, E. Stensrud, B. Kitchenham, I. Myrtveit, A simulation study of the model evaluation criterion MMRE, Softw. Eng. IEEE Trans. 29 (2003) 985–995. 28. I. (Norwegian S. of M. Myrtveit, E. (Buskerud U.C. Stensrud, M. (Bournemouth U. Shepperd, Reliability and validity in comparative studies of software prediction models, IEEE Trans. Softw. Eng. 31 (2005) 380–391. doi:10.1109/TSE.2005.58. 29. M. Shepperd, S. MacDonell, Evaluating prediction systems in software project estimation, Inf. Softw. Technol. 54 (2012) 820–827. doi:10.1016/j.infsof.2011.12.008. 30. L. Lavazza, S. Morasca, On the Evaluation of Effort Estimation Models, in: Proc. 21st Int. Conf. Eval. Assess. Softw. Eng. - EASE’17, Karlskrona, Sweden, 2017: pp. 41–50. doi:10.1145/3084226.3084260. 31. J. Demšar, Statistical Comparisons of Classifiers over Multiple Data Sets, J. Mach. Learn. Res. 7 (2006) 1–30. 32. S. García, A. Fernández, J. Luengo, F. Herrera, Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power, Inf. Sci. (Ny). 180 (2010) 2044–2064. doi:10.1016/j.ins.2009.12.010. 90 F. Valdés-Souto 33. O.J.D. Source, Multiple Comparisons Among Means, J. Am. Stat. Assoc. 56 (1961) 52–64. 91 IWSM/Mensura’18, September 18–20, 2018, Beijing, China Presence (level, Appendix I. Data Provided by Participants not num- for Each Use Case Identified ber) of objects Table A1 shows the data provided by partici- of inter- pants for each Functional Process identified in the Use Case size est re- experiment. Project (value assign- lated to UC ID - Column 1 presents the Project identifier. For ID ment – range the Use confidentially purposes, the Projects were la- from 0 - 5) Case beled sequentially, from “Proj 1” to “Proj 4. (value assign- - Column 2 presents the Use Case identifier. For ment – confidentially purposes, the Use Cases were la- range beled sequentially, from “UC 1” to “UC 293. from 0 - - Column 3 presents the functional size obtained 5) utilizing the standard COSMIC method – in CFP units, Proj 1 UC 9 3.5 3 - Column 4 presents the value assigned for the in- Proj 1 UC 10 3.5 3.5 put variable “Use Case size” for the EPCU ap- proximation technique. Proj 1 UC 11 3.5 2.95 - Column 5 presents the value assigned for the in- put variable “Presence of objects of interest re- Proj 1 UC 12 3.5 3 lated to the Use Cases” for the EPCU approxi- Proj 1 UC 13 3.5 2.75 mation technique. Proj 1 UC 14 4 2.5 Table A1: Data collected by participants (project lead- ers) Proj 1 UC 15 3.5 3.65 Presence Proj 1 UC 16 3 4 (level, not num- Proj 1 UC 17 3.5 3 ber) of objects Proj 1 UC 18 3 3 of inter- Proj 1 UC 19 3 3 Use Case size est re- Project (value assign- lated to Proj 1 UC 20 3.5 3.5 UC ID ID ment – range the Use from 0 - 5) Case Proj 1 UC 21 3.5 3.5 (value assign- Proj 1 UC 22 3.5 3.5 ment – range Proj 1 UC 23 4 3.5 from 0 - 5) Proj 1 UC 24 4 3.5 Proj 1 UC 1 3 2.5 Proj 1 UC 2 3 3 Proj 1 UC 25 3.5 3.5 Proj 1 UC 3 3 3 Proj 1 UC 26 4 3.5 Proj 1 UC 4 3 2.5 Proj 1 UC 5 3.5 3 Proj 1 UC 27 3.5 3.5 Proj 1 UC 6 3 3 Proj 1 UC 28 4 3 Proj 1 UC 7 3.5 3.5 Proj 1 UC 29 3.5 3 Proj 1 UC 8 3.5 3 Proj 1 UC 30 3.5 3 Proj 1 UC 31 3.5 3 92 F. Valdés-Souto Presence Presence (level, (level, not num- not num- ber) of ber) of objects objects of inter- of inter- Use Case size est re- Use Case size est re- Project (value assign- lated to Project (value assign- lated to UC ID UC ID ID ment – range the Use ID ment – range the Use from 0 - 5) Case from 0 - 5) Case (value (value assign- assign- ment – ment – range range from 0 - from 0 - 5) 5) Proj 1 UC 32 4 3.5 Proj 2 UC 68 2.35 2.35 Proj 2 UC 69 2.55 2.1 Proj 1 UC 33 4 3 Proj 2 UC 70 2.55 2.1 Proj 1 UC 34 4 3 Proj 2 UC 71 2.6 2.6 Proj 2 UC 72 2.5 1.4 Proj 1 UC 35 4 3.5 Proj 2 UC 73 2.4 2.6 Proj 1 UC 36 4 3.5 Proj 2 UC 74 1.7 1.95 Proj 1 UC 37 4 4 Proj 2 UC 75 2.1 2.1 Proj 1 UC 38 3.85 3.5 Proj 2 UC 76 2.3 1.95 Proj 1 UC 39 4 3.5 Proj 2 UC 77 1.9 1.6 Proj 1 UC 40 4 3 Proj 2 UC 78 2.3 2.3 Proj 1 UC 41 4 4 Proj 2 UC 79 2.45 2.6 Proj 1 UC 42 4 4 Proj 2 UC 80 2.35 2.15 Proj 1 UC 43 4 3.5 Proj 2 UC 81 2.8 2.6 Proj 2 UC 44 2.4 1.8 Proj 2 UC 82 2.9 2.05 Proj 2 UC 45 2.5 2.05 Proj 2 UC 83 2.5 1.75 Proj 2 UC 46 2.85 2.4 Proj 2 UC 84 2.1 1.95 Proj 2 UC 47 2.3 2.2 Proj 2 UC 85 2.45 2.1 Proj 2 UC 48 3 2.6 Proj 2 UC 86 3.05 1.6 Proj 2 UC 49 2.1 1.85 Proj 2 UC 87 2.35 1.6 Proj 2 UC 50 2.55 2.55 Proj 2 UC 88 3 3 Proj 2 UC 51 2.65 2.45 Proj 2 UC 89 2.1 1.95 Proj 2 UC 52 2.35 2.8 Proj 2 UC 90 1.95 1.95 Proj 2 UC 53 1.65 1.8 Proj 2 UC 91 1.95 1.95 Proj 2 UC 54 2.8 2.8 Proj 2 UC 92 1.8 1.6 Proj 2 UC 55 2.85 2.55 Proj 2 UC 93 2 2.15 Proj 2 UC 56 2.3 2.1 Proj 2 UC 94 2.3 1.95 Proj 2 UC 57 2.6 2.6 Proj 2 UC 95 2.5 2.15 Proj 2 UC 58 1.95 1.95 Proj 2 UC 96 2.45 2.25 Proj 2 UC 59 2.3 2.1 Proj 2 UC 97 2.6 2.6 Proj 2 UC 60 2.65 2.1 Proj 2 UC 98 2.5 2.55 Proj 2 UC 61 2.1 2.05 Proj 2 UC 99 2.5 2.15 Proj 2 UC 62 3.1 2.65 Proj 2 UC 100 3.25 2.2 Proj 2 UC 63 2.5 2.1 Proj 2 UC 101 2.15 1.8 Proj 2 UC 64 2.6 2.25 Proj 2 UC 102 2.25 2.05 Proj 2 UC 65 2.5 2.5 Proj 2 UC 103 2.3 2.1 Proj 2 UC 66 2.35 2.3 Proj 2 UC 104 2.6 1.85 Proj 2 UC 67 2.35 2.1 Proj 2 UC 105 2.65 2.3 93 IWSM/Mensura’18, September 18–20, 2018, Beijing, China Presence Presence (level, (level, not num- not num- ber) of ber) of objects objects of inter- of inter- Use Case size est re- Use Case size est re- Project (value assign- lated to Project (value assign- lated to UC ID UC ID ID ment – range the Use ID ment – range the Use from 0 - 5) Case from 0 - 5) Case (value (value assign- assign- ment – ment – range range from 0 - from 0 - 5) 5) Proj 2 UC 106 2.8 1.75 Proj 2 UC 137 2.35 2 Proj 2 UC 107 2.75 2.35 Proj 2 UC 138 3.25 1.75 Proj 2 UC 108 2.7 2.55 Proj 2 UC 139 2.6 2.05 Proj 2 UC 109 2.9 2.6 Proj 3 UC 140 3.5 3.75 Proj 2 UC 110 2.85 2.3 Proj 3 UC 141 3.5 3 Proj 2 UC 111 2.85 2.6 Proj 3 UC 142 3.5 2.5 Proj 2 UC 112 2.75 2.55 Proj 3 UC 143 3.25 3.6 Proj 2 UC 113 2.4 2.2 Proj 3 UC 144 3.3 3.4 Proj 2 UC 114 2.5 1.65 Proj 3 UC 145 2.15 3 Proj 2 UC 115 2.05 1.65 Proj 3 UC 146 3.6 3.9 Proj 2 UC 116 2.5 2.5 Proj 3 UC 147 3.5 3.65 Proj 2 UC 117 2.35 1.65 Proj 3 UC 148 2.75 3.05 Proj 2 UC 118 2.55 2.15 Proj 3 UC 149 3.25 3.35 Proj 2 UC 119 1.8 1.75 Proj 3 UC 150 3 4 Proj 2 UC 120 2.3 1.8 Proj 3 UC 151 3.15 3 Proj 2 UC 121 2.1 1.75 Proj 3 UC 152 2.55 3.8 Proj 2 UC 122 2.75 1.75 Proj 3 UC 153 2.6 3 Proj 2 UC 123 2.95 2.2 Proj 3 UC 154 2.55 2.5 Proj 2 UC 124 2.75 2.25 Proj 3 UC 155 3 3.75 Proj 2 UC 125 3.05 2.65 Proj 3 UC 156 2.25 2.75 Proj 3 UC 157 2.25 2.75 Proj 2 UC 126 2.65 1.75 Proj 3 UC 158 3 3 Proj 3 UC 159 2.5 3.25 Proj 2 UC 127 2.7 1.75 Proj 3 UC 160 1.5 3 Proj 2 UC 128 2.5 1.75 Proj 3 UC 161 2.7 2.9 Proj 2 UC 129 2.1 1.75 Proj 3 UC 162 2.45 2.5 Proj 2 UC 130 1.8 1.75 Proj 3 UC 163 2.65 3.25 Proj 2 UC 131 2.55 1.75 Proj 3 UC 164 1.8 2.75 Proj 2 UC 132 2.8 1.75 Proj 3 UC 165 2.65 3.5 Proj 2 UC 133 2.55 2 Proj 3 UC 166 2.85 3.75 Proj 2 UC 134 2.35 1.75 Proj 3 UC 167 1.4 2.75 Proj 2 UC 135 2.55 2.2 Proj 3 UC 168 3 3.5 Proj 2 UC 136 2.3 1.7 94 F. Valdés-Souto Presence Presence (level, (level, not num- not num- ber) of ber) of objects objects of inter- of inter- Use Case size est re- Use Case size est re- Project (value assign- lated to Project (value assign- lated to UC ID UC ID ID ment – range the Use ID ment – range the Use from 0 - 5) Case from 0 - 5) Case (value (value assign- assign- ment – ment – range range from 0 - from 0 - 5) 5) Proj 3 UC 169 3 2.65 Proj 4 UC 207 2.8 2.6 Proj 3 UC 170 3 3.5 Proj 4 UC 208 3.4 2.8 Proj 3 UC 171 2.95 3.25 Proj 4 UC 209 3 2.8 Proj 3 UC 172 3.25 3.5 Proj 4 UC 210 3.2 2.2 Proj 3 UC 173 3.25 3.25 Proj 4 UC 211 3.3 2.5 Proj 3 UC 174 2.95 4.25 Proj 4 UC 212 2.7 2.8 Proj 3 UC 175 3.05 3.5 Proj 4 UC 213 3.2 2.3 Proj 3 UC 176 2.8 2.35 Proj 4 UC 214 3.2 3 Proj 3 UC 177 2.3 3 Proj 4 UC 215 2.5 2.8 Proj 3 UC 178 2.9 2.35 Proj 4 UC 216 2.8 2.4 Proj 3 UC 179 2.2 2.65 Proj 4 UC 217 2.1 2 Proj 3 UC 180 2.55 2.35 Proj 4 UC 218 3 2.6 Proj 3 UC 181 2.3 2.3 Proj 4 UC 219 3.1 3 Proj 3 UC 182 2.35 2.15 Proj 4 UC 220 2.8 3 Proj 3 UC 183 2.45 2.2 Proj 4 UC 221 3.2 3.1 Proj 3 UC 184 2.35 2.5 Proj 4 UC 222 1.9 2.3 Proj 3 UC 185 2.15 2.05 Proj 4 UC 223 2.6 3 Proj 3 UC 186 2.5 3.85 Proj 4 UC 224 3 2.5 Proj 3 UC 187 2.7 4 Proj 4 UC 225 3.3 2.4 Proj 3 UC 188 2.1 3 Proj 4 UC 226 3 3.1 Proj 3 UC 189 2.55 3.25 Proj 4 UC 227 2.5 2.5 Proj 3 UC 190 2.8 3 Proj 4 UC 228 3.5 2.5 Proj 3 UC 191 2.6 3.5 Proj 4 UC 229 2.5 2.5 Proj 3 UC 192 3.55 3.85 Proj 4 UC 230 3 3 Proj 3 UC 193 3.05 3.45 Proj 4 UC 231 3 2.5 Proj 3 UC 194 2.8 3.35 Proj 4 UC 232 3.2 2.5 Proj 4 UC 195 3.5 3.3 Proj 4 UC 233 3.3 3 Proj 4 UC 196 3.4 3.4 Proj 4 UC 234 3.2 3 Proj 4 UC 197 3 2.8 Proj 4 UC 235 3.4 3.1 Proj 4 UC 198 3 3.1 Proj 4 UC 236 3.2 2.5 Proj 4 UC 199 3.3 3.4 Proj 4 UC 237 2.5 3 Proj 4 UC 200 3.3 3.5 Proj 4 UC 238 3.9 3 Proj 4 UC 201 3.6 3 Proj 4 UC 239 3.5 2.9 Proj 4 UC 202 3.1 2.5 Proj 4 UC 240 3.2 3.4 Proj 4 UC 203 3.4 3 Proj 4 UC 204 3.3 3 Proj 4 UC 241 2.7 2.7 Proj 4 UC 205 3.3 3 Proj 4 UC 242 3.5 3.1 Proj 4 UC 206 2.6 2.5 Proj 4 UC 243 3 2.3 95 IWSM/Mensura’18, September 18–20, 2018, Beijing, China Presence Presence (level, (level, not num- not num- ber) of ber) of objects objects of inter- of inter- Use Case size est re- Use Case size est re- Project (value assign- lated to Project (value assign- lated to UC ID UC ID ID ment – range the Use ID ment – range the Use from 0 - 5) Case from 0 - 5) Case (value (value assign- assign- ment – ment – range range from 0 - from 0 - 5) 5) Proj 4 UC 244 3 2.5 Proj 4 UC 275 2.7 2.7 Proj 4 UC 245 2.5 2.5 Proj 4 UC 276 2 2.5 Proj 4 UC 246 2.7 2.4 Proj 4 UC 277 3 2.6 Proj 4 UC 247 3.3 3 Proj 4 UC 278 3.1 2.6 Proj 4 UC 248 2.7 2 Proj 4 UC 279 3.2 1.8 Proj 4 UC 249 3.8 3.5 Proj 4 UC 280 3 3 Proj 4 UC 250 3.3 3.1 Proj 4 UC 281 3.2 2.8 Proj 4 UC 251 3.1 2.5 Proj 4 UC 252 2.8 2.8 Proj 4 UC 282 3 2.5 Proj 4 UC 253 3.6 3.5 Proj 4 UC 254 2.5 2.6 Proj 4 UC 283 2 2 Proj 4 UC 255 2.9 2.6 Proj 4 UC 256 3.3 2.5 Proj 4 UC 284 3.3 2.4 Proj 4 UC 257 2.6 2 Proj 4 UC 258 3.2 2.4 Proj 4 UC 285 3 2.9 Proj 4 UC 259 2.5 2.3 Proj 4 UC 286 3 2.3 Proj 4 UC 260 2.5 2.2 Proj 4 UC 287 2.5 1.8 Proj 4 UC 261 3 2.9 Proj 4 UC 288 3 2.5 Proj 4 UC 262 2.7 3 Proj 4 UC 289 2.5 2 Proj 4 UC 263 2.7 2.8 Proj 4 UC 290 3.3 2.6 Proj 4 UC 264 3.2 3 Proj 4 UC 291 3.9 2 Proj 4 UC 265 2.5 2.2 Proj 4 UC 292 2 2 Proj 4 UC 266 2.5 2.2 Proj 4 UC 267 3.8 3 Proj 4 UC 293 2.5 2.5 Proj 4 UC 268 3.4 2.8 Proj 4 UC 269 3.6 3 Proj 4 UC 270 3.6 3.5 Proj 4 UC 271 3.2 2.7 Proj 4 UC 272 2.4 2.9 Proj 4 UC 273 2.5 1.8 Proj 4 UC 274 2.7 2.7 96 F. Valdés-Souto Appendix II. COSMIC Functional Size and Pro- UC RE EPCU1 EPC Approximation ject ESB ID AL 6.4 U44 ID COSMIC functional size and approxi- mation for each Functional Process are pre- sented in Table A2 II where: 14.6 1 UC 4 22 7.7 9.84 - Column 1 presents the Project identifier. 0 For purposes of confidentiality , the Projects 13 26.7 1 UC 5 10.7 12.56 were labeled sequentially, from “Proj 1” to 2 8 “Proj 4, 27.5 1 UC 6 9 7.7 12.72 - Column 2 presents the Use Case identifier. 2 For purposes of confidentiality, the Use 1 UC 7 4 10.7 14.72 36.4 Cases were labeled sequentially, from “UC 9 1” to “UC 293, 34 26.7 1 UC 8 10.7 12.56 3 8 - Column 3 presents the functional size ob- tained utilizing the standard COSMIC 26.7 1 UC 9 11 10.7 12.56 method – in CFP units, 8 UC 36.4 - Column 4 presents the Equal Size Bands ap- 1 8 10.7 14.72 10 9 proximation approach, UC 26.7 - Column 5 presents the EPCU size approxi- 1 20 10.7 12.56 11 8 mation approach using an output variable UC 26.7 domain function from 2 - 16.4 CFP [9] [10], 1 12 8 10.7 12.56 8 and UC 22.9 1 6 10.7 11.71 - Column 6 presents the EPCU size approxi- 13 6 mation approach using an output variable domain function from 2 - 44 CFP [11]. 1 UC 14 154 10.7 9.84 14.60 1 UC 15 12 10.7 15.98 42.11 1 UC 16 9 7.7 16.40 44.00 1 UC 17 37 10.7 12.56 26.78 1 UC 18 75 7.7 12.72 27.52 1 UC 19 13 7.7 12.72 27.52 1 UC 20 9 10.7 14.72 36.49 Table A2: Functional size – Real and from 3 approximation techniques 1 UC 21 26 10.7 14.72 36.49 1 UC 22 132 10.7 14.72 36.49 Pro- UC RE EPCU1 EPC 1 UC 23 14 10.7 15.09 38.12 ject ESB ID AL 6.4 U44 ID 1 UC 24 9 10.7 15.09 38.12 1 UC 25 37 10.7 14.72 36.49 14.6 1 UC 1 9 7.7 9.84 1 UC 26 49 10.7 15.09 38.12 0 27.5 1 UC 2 9 7.7 12.72 1 UC 27 11 10.7 14.72 36.49 2 27.5 1 UC 28 9 10.7 12.46 26.36 1 UC 3 9 7.7 12.72 2 1 UC 29 22 10.7 12.56 26.78 97 IWSM/Mensura’18, September 18–20, 2018, Beijing, China Pro- Pro- UC RE EPCU1 EPC UC RE EPCU1 EPC ject ESB ject ESB ID AL 6.4 U44 ID AL 6.4 U44 ID ID 1 UC 30 16 10.7 12.56 26.78 2 UC 68 17 7.7 8.92 12.97 1 UC 31 8 10.7 12.56 26.78 2 UC 69 9 7.7 8.77 12.36 1 UC 32 8 10.7 15.09 38.12 2 UC 70 19 7.7 8.77 12.36 2 UC 71 10 7.7 10.56 17.81 1 UC 33 21 10.7 12.46 26.36 2 UC 72 12 7.7 6.95 8.53 1 UC 34 19 10.7 12.46 26.36 2 UC 73 19 7.7 10.08 16.26 2 UC 74 17 4.8 6.02 7.74 1 UC 35 32 10.7 15.09 38.12 2 UC 75 9 7.7 7.01 9.60 1 UC 36 5 10.7 15.09 38.12 2 UC 76 12 7.7 7.55 10.36 1 UC 37 46 10.7 16.40 44.00 2 UC 77 20 4.8 5.49 6.98 1 UC 38 17 10.7 15.09 38.12 2 UC 78 13 7.7 8.16 11.64 1 UC 39 11 10.7 15.09 38.12 2 UC 79 15 7.7 10.36 16.95 1 UC 40 15 10.7 12.46 26.36 2 UC 80 22 7.7 8.48 12.05 1 UC 41 15 10.7 16.40 44.00 2 UC 81 37 7.7 10.66 18.25 1 UC 42 16 10.7 16.40 44.00 2 UC 82 19 7.7 8.61 12.01 1 UC 43 10 10.7 15.09 38.12 2 UC 83 14 7.7 8.00 10.74 2 UC 44 4 7.7 7.53 10.05 2 UC 84 9 7.7 6.83 9.24 2 UC 45 27 7.7 8.79 12.39 2 UC 85 7 7.7 8.79 12.39 2 UC 46 31 7.7 9.41 13.70 2 UC 86 5 10.7 7.74 10.18 2 UC 47 39 7.7 7.97 11.24 2 UC 87 14 7.7 7.04 9.03 2 UC 48 37 7.7 10.73 18.59 2 UC 88 8 7.7 12.72 27.52 2 UC 49 8 7.7 6.65 8.86 2 UC 89 31 7.7 6.83 9.24 2 UC 50 6 7.7 10.56 17.81 2 UC 90 21 4.8 6.56 8.80 2 UC 51 20 7.7 9.84 14.60 2 UC 91 11 4.8 6.56 8.80 2 UC 52 47 7.7 11.05 20.60 2 UC 92 4 4.8 5.21 6.63 2 UC 53 20 4.8 5.50 7.09 2 UC 93 27 4.8 7.16 9.76 2 UC 54 12 7.7 11.80 23.38 2 UC 94 31 7.7 7.55 10.36 2 UC 55 23 7.7 10.70 18.45 2 UC 95 39 7.7 9.05 12.94 2 UC 56 32 7.7 7.76 10.81 2 UC 96 37 7.7 9.32 13.50 2 UC 57 32 7.7 10.56 17.81 2 UC 97 8 7.7 10.56 17.81 2 UC 58 11 4.8 6.56 8.80 2 UC 98 6 7.7 10.36 16.95 2 UC 59 35 7.7 7.76 10.81 2 UC 99 20 7.7 9.05 12.94 2 UC 60 59 7.7 8.73 12.26 UC 2 47 10.7 8.80 12.42 2 UC 61 23 7.7 7.01 9.60 100 2 UC 62 39 10.7 11.43 21.72 UC 2 20 7.7 6.76 8.93 2 UC 63 52 7.7 8.79 12.39 101 2 UC 64 9 7.7 9.27 13.39 UC 2 23 7.7 7.76 10.81 102 2 UC 65 9 7.7 9.84 14.60 UC 2 UC 66 6 7.7 8.71 12.53 2 32 7.7 7.76 10.81 103 2 UC 67 8 7.7 8.25 11.56 98 F. Valdés-Souto Pro- Pro- UC RE EPCU1 EPC UC RE EPCU1 EPC ject ESB ject ESB ID AL 6.4 U44 ID AL 6.4 U44 ID ID UC UC 2 32 7.7 8.26 11.28 2 37 7.7 8.00 10.74 104 128 UC UC 2 35 7.7 9.15 13.16 2 9 7.7 6.47 8.48 105 129 UC UC 2 59 7.7 8.04 10.81 2 14 4.8 5.75 7.48 106 130 UC UC 2 23 7.7 9.43 13.74 2 7 7.7 8.01 10.75 107 131 UC UC 2 39 7.7 10.61 18.04 2 5 7.7 8.04 10.81 108 132 UC UC 2 52 7.7 10.70 18.45 2 8 7.7 8.52 11.82 109 133 UC UC 2 9 7.7 9.08 13.00 2 14 7.7 7.53 10.05 110 134 UC UC 2 9 7.7 10.70 18.45 2 31 7.7 9.02 12.88 111 135 UC UC 2 6 7.7 10.66 18.25 2 21 7.7 6.89 8.98 112 136 UC UC 2 8 7.7 8.48 12.05 2 11 7.7 8.01 11.06 113 137 UC UC 2 17 7.7 7.74 10.18 2 12 10.7 8.07 10.88 114 138 UC UC 2 9 7.7 6.28 8.09 2 15 7.7 8.77 12.36 115 139 UC UC 2 19 7.7 9.84 14.60 3 121 10.7 16.40 44.00 116 140 UC UC 2 10 7.7 7.29 9.54 3 8 10.7 12.56 26.78 117 141 UC UC 2 12 7.7 9.02 12.88 3 15 10.7 9.84 14.60 118 142 UC UC 2 19 4.8 5.75 7.48 3 5 10.7 15.17 38.48 119 143 UC UC 2 17 7.7 7.11 9.45 3 124 10.7 14.10 33.70 120 144 UC UC 2 9 7.7 6.47 8.48 3 13 7.7 11.27 22.46 121 145 UC UC 2 12 7.7 8.04 10.81 3 132 10.7 16.40 44.00 122 146 UC UC 2 20 7.7 8.79 12.39 3 153 10.7 15.98 42.11 123 147 UC UC 2 13 7.7 9.11 13.07 3 22 7.7 13.01 28.82 124 148 UC UC 2 15 10.7 11.43 21.72 3 22 10.7 14.10 33.70 125 149 UC UC 2 19 7.7 8.02 10.77 3 22 7.7 16.40 44.00 126 150 UC UC 2 22 7.7 8.02 10.77 3 6 10.7 12.74 27.61 127 151 99 IWSM/Mensura’18, September 18–20, 2018, Beijing, China Pro- Pro- UC RE EPCU1 EPC UC RE EPCU1 EPC ject ESB ject ESB ID AL 6.4 U44 ID AL 6.4 U44 ID ID UC UC 3 103 7.7 16.40 44.00 3 23 7.7 16.40 44.00 152 174 UC UC 3 29 7.7 12.48 26.44 3 30 10.7 14.53 35.60 153 175 UC UC 3 17 7.7 9.84 14.60 3 39 7.7 9.43 13.74 154 176 UC UC 3 13 7.7 16.40 44.00 3 16 7.7 11.62 23.62 155 177 UC UC 3 20 7.7 10.73 19.66 3 19 7.7 9.41 13.70 156 178 UC UC 3 13 7.7 10.73 19.66 3 17 7.7 10.06 17.19 157 179 UC UC 3 15 7.7 12.72 27.52 3 5 7.7 9.48 13.85 158 180 UC UC 3 11 7.7 14.04 33.42 3 2 7.70 8.16 11.64 159 181 UC UC 3 15 4.8 9.31 16.00 3 5 7.70 8.48 12.05 160 182 UC UC 3 10 7.7 12.06 24.56 3 82 7.70 9.05 12.94 161 183 UC UC 3 54 7.7 9.84 14.60 3 62 7.70 9.58 14.05 162 184 UC UC 3 9 7.7 13.95 33.01 3 29 7.70 7.36 10.15 163 185 UC UC 3 40 4.8 9.64 17.05 3 40 7.70 16.40 44.00 164 186 UC UC 3 35 7.7 14.83 36.98 3 41 7.70 16.40 44.00 165 187 UC UC 3 23 7.7 16.40 44.00 3 21 7.70 10.96 21.42 166 188 UC UC 3 15 4.8 8.35 13.11 3 14 7.70 14.01 33.31 167 189 UC UC 3 47 7.7 14.55 35.72 3 36 7.70 12.59 26.93 168 190 UC UC 3 11 7.7 11.41 21.62 3 36 7.70 15.01 37.79 169 191 UC UC 3 26 7.7 14.55 35.72 3 12 10.70 16.40 44.00 170 192 UC UC 3 31 7.7 13.69 31.83 3 55 10.70 14.53 35.60 171 193 UC UC 3 15 10.7 14.58 35.82 3 18 7.70 14.25 34.39 172 194 UC UC 3 29 10.7 13.71 31.96 4 37 10.70 13.90 32.81 173 195 100 F. Valdés-Souto Pro- Pro- UC RE EPCU1 EPC UC RE EPCU1 EPC ject ESB ject ESB ID AL 6.4 U44 ID AL 6.4 U44 ID ID UC UC 4 33 10.70 14.18 34.04 4 78 7.70 12.59 26.93 196 220 UC UC 4 9 7.70 11.94 24.01 4 35 10.70 13.05 28.97 197 221 UC UC 4 31 7.70 13.04 28.94 4 17 4.80 7.33 9.91 198 222 UC UC 4 32 10.70 14.10 33.70 4 83 7.70 12.48 26.44 199 223 UC UC 4 49 10.70 14.58 35.82 4 12 7.70 9.84 14.60 200 224 UC UC 4 64 10.70 12.50 26.53 4 5 10.70 9.40 13.68 201 225 UC UC 4 10 10.70 9.84 14.60 4 8 7.70 13.04 28.94 202 226 UC UC 4 18 10.70 12.62 27.08 4 6 7.70 9.84 14.60 203 227 UC UC 4 15 10.70 12.69 27.38 4 91 10.70 9.84 14.60 204 228 UC UC 4 31 10.70 12.69 27.38 4 26 7.70 9.84 14.60 205 229 UC UC 4 20 7.70 9.84 14.60 4 22 7.70 12.72 27.52 206 230 UC UC 4 13 7.70 10.66 18.25 4 158 7.70 9.84 14.60 207 231 UC UC 4 42 10.70 11.84 23.57 4 7 10.70 9.84 14.60 208 232 UC UC 4 64 7.70 11.94 24.01 4 21 10.70 12.69 27.38 209 233 UC UC 4 50 10.70 8.78 12.37 4 3 10.70 12.74 27.61 210 234 UC UC 4 24 10.70 9.84 14.60 4 15 10.70 13.02 28.85 211 235 UC UC 4 17 7.70 11.62 22.56 4 6 10.70 9.84 14.60 212 236 UC UC 4 7 10.70 9.05 12.94 4 4 7.70 12.46 26.36 213 237 UC UC 4 3 10.70 12.74 27.61 4 24 10.70 12.46 26.36 214 238 UC UC 4 3 7.70 11.41 21.66 4 5 10.70 12.12 24.81 215 239 UC UC 4 6 7.70 9.43 13.74 4 13 10.70 14.06 33.49 216 240 UC UC 4 7 7.70 6.83 9.24 4 11 7.70 11.21 20.75 217 241 UC UC 4 2 7.70 10.73 18.59 4 11 10.70 13.01 28.79 218 242 UC UC 4 23 10.70 12.75 27.66 4 4 7.70 9.06 12.95 219 243 101 IWSM/Mensura’18, September 18–20, 2018, Beijing, China Pro- Pro- UC RE EPCU1 EPC UC RE EPCU1 EPC ject ESB ject ESB ID AL 6.4 U44 ID AL 6.4 U44 ID ID UC UC 4 39 7.70 9.84 14.60 4 4 7.70 9.05 12.94 244 266 UC UC 4 18 7.70 9.84 14.60 4 14 10.70 12.46 26.36 245 267 UC UC 4 26 7.70 9.46 13.79 4 13 10.70 11.84 23.57 246 268 UC UC 4 8 10.70 12.69 27.38 4 16 10.70 12.50 26.53 247 269 UC UC 4 8 7.70 8.50 11.77 4 14 10.70 14.93 37.42 248 270 UC UC 4 23 10.70 15.09 38.12 4 16 10.70 11.42 21.69 249 271 UC UC 4 4 10.70 13.03 28.91 4 15 7.70 11.53 22.76 250 272 UC UC 4 16 10.70 9.84 14.60 4 16 7.70 8.00 10.74 251 273 UC UC 4 32 7.70 11.80 23.38 4 6 7.70 11.21 20.75 252 274 UC UC 4 45 10.70 14.93 37.42 4 5 7.70 11.21 20.75 253 275 UC UC 4 7 7.70 10.36 16.95 4 6 4.80 8.53 11.84 254 276 UC UC 4 21 7.70 10.70 18.45 4 3 7.70 10.73 18.59 255 277 UC UC 4 27 10.70 9.84 14.60 4 3 10.70 10.74 18.65 256 278 UC UC 4 15 7.70 8.52 11.82 4 28 10.70 8.09 10.92 257 279 UC UC 4 14 10.70 9.39 13.65 4 6 7.70 12.72 27.52 258 280 UC UC 4 6 7.70 9.32 13.50 4 6 10.70 11.96 24.09 259 281 UC 4 14 7.70 9.05 12.94 UC 260 4 9 7.70 9.84 14.60 UC 282 4 3 7.70 12.37 25.94 261 UC UC 4 3 4.80 6.56 8.80 4 7 7.70 12.53 26.64 283 262 UC UC 4 3 7.70 11.62 22.56 4 19 10.70 9.40 13.68 263 284 UC 4 15 10.70 12.74 27.61 UC 264 4 7 7.70 12.37 25.94 285 UC 4 2 7.70 9.05 12.94 265 102 F. Valdés-Souto Pro- UC RE EPCU1 EPC ject ESB ID AL 6.4 U44 ID UC 4 6 7.70 9.06 12.95 286 UC 4 22 7.70 8.00 10.74 287 UC 4 3 7.70 9.84 14.60 288 UC 4 60 7.70 8.53 11.84 289 UC 4 16 10.70 10.72 18.53 290 UC 4 2 10.70 8.53 11.84 291 UC 4 108 4.80 6.56 8.80 292 UC 4 4 7.70 9.84 14.60 293 103 IWSM/Mensura’18, September 18–20, 2018, Beijing, China Appendix III. Friedman Test Results from SPSS® Descriptive Statistics Std. Devia- Maxi- N Mean tion Minimun mun REAL 293 24.7406 31.84897 0.00 343.00 ESB 293 8.5556 1.68440 4.80 10.70 EPCU16 293 10.8944 2.76991 5.21 16.40 EPCU44 293 21.0759 10.40606 6.63 44.00 Ranks Mean Rank REAL 2.89 ESB 1.35 EPCU16 2.20 EPCU44 3.55 Tests Statisticsa N 293 Chi-Square 468.936 df 3 Asymp. Sig. 2.57215388100136E- 101 104 F. Valdés-Souto Appendix IV. Wilcoxon Test Results from SPSS® ESB – REAL N Mean Rank Sum of Ranks ESB - REAL Negative Ranks 230a 165.49 38063.00 Positive Ranks 63b 79.49 5008.00 Ties 0c Total 293 a. ESB < REAL b. ESB > REAL c. ESB = REAL Test Statisticsa ESB - REAL Z -11.388b Asymp. 4.8089036753386E- Sig. (2-tai- 30 led) a. Wilcoxon Test with sign b. Based in negative ranks. EPCU16 - REAL Ranks N Mean Rank Sum of Ranks EPCU16 - REAL Negative Ranks 194 a 177.95 34523.00 Positive Ranks 99 b 86.34 8548.00 Ties 0c Total 293 a. EPCU16 < REAL 105 IWSM/Mensura’18, September 18–20, 2018, Beijing, China b. EPCU16 > REAL c. EPCU16 = REAL Test Statisticsa EPCU16 - REAL Z -8.948b Asymp. Sig. 3.6339E-19 (2-tailed) a. Wilcoxon Test with sign b. Based in positive ranks. EPCU44 – REAL Ranks N Mean Rank Sum of Ranks EPCU44 - REAL Negative Ranks a 130 153.62 19971.00 Positive Ranks b 163 141.72 23100.00 Ties 0 c Total 293 a. EPCU44 < REAL b. EPCU44 > REAL c. EPCU44 = REAL Test Statisticsa EPCU44 - REAL Z -1.078b Asymp. Sig. (2- .281 tailed) a. Wilcoxon Test with sign b. Based in positive ranks. 106 F. Valdés-Souto EPCU44 – ESB Ranks N Mean Rank Sum of Ranks EPCU44 - ESB Negative Ranks 1a 4.00 4.00 Positive Ranks 292b 147.49 43067.00 Ties 0 c Total 293 a. EPCU44 < ESB b. EPCU44 > ESB c. EPCU44 = ESB Tests Statisticsa MRE_EPCU44 - ESB Z -14.835b Asymp. Sig. (2-tailed) 8.7473E-50 a. Wilcoxon Test with sign b. Based in positive ranks. EPCU16 – ESB Ranks N Mean Rank Sum of Ranks EPCU16 - ESB Negative Ranks 39a 68.58 2674.50 Positive Ranks 254b 159.04 40396.50 Ties 0 c Total 293 a. EPCU16 < ESB b. EPCU16 > ESB c. EPCU16 = ESB 107 IWSM/Mensura’18, September 18–20, 2018, Beijing, China Test Statisticsa MRE_EPCU16 - ESB Z -12.995b Asymp. Sig. (2-tai- 1.3091E-38 led) a. Wilcoxon Test with sign b. Based in positive ranks. EPCU16 - EPCU44 Ranks N Mean Rank Sum of Ranks EPCU16 - Negative Ranks 0a 0.00 0.00 EPCU44 Positive Ranks 293b 147.00 43071.00 Ties 0c Total 293 a. EPCU16 < EPCU44 b. EPCU16 > EPCU44 c. EPCU16 = EPCU44 Test Statisticsa MRE_EPCU16 - MRE_EPCU44 Z -14.839b Asymp. Sig. (2-tai- 8.2436E-50 led) 108