5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2017) Estimating Web Service Quality of Service Parameters using Source Code Metrics and LSSVM Lov Kumar Santanu Rath Ashish Sureka NIT Rourkela, India NIT Rourkela, India Ashoka University, India lovkumar505@gmail.com skrath@nitrkl.ac.in ashish.sureka@ashoka.edu.in Abstract—We conduct an empirical analysis to investigate the a model using LSSVM method with three different types of relationship between thirty seven different source code metrics kernel functions: linear kernel, polynomial kernel and RBF with fifteen different Web Service QoS (Quality of Service) kernel. LSSVM method is a least square version of support parameters. The source code metrics used in our experiments consists of nineteen Object-Oriented metrics, six Baski and vector machine (SVM) and is based on statistical learning Misra metrics, and twelve Harry M. Sneed metrics. We apply theory [18]. Principal Component Analysis (PCA) and Rough Set Analysis for The overall effectiveness and performance of the QoS feature extraction and selection. The different sets of metrics are parameter prediction models depends on the subset of source provided as input to the predictive model generated using Least code metrics used as input to develop the statistical models. Square Support Vector Machine (LSSVM) with three different types of kernel functions: RBF, Polynomial, and Linear. Our In our work, six different sets of source code metrics: all experimental results reveal that the prediction model developed metrics (AM) for source code (thirty seven metrics), Baski using LSSVM method with RBF kernel function is more effective and Misra Metrics suite (BMS), Harry M. Sneed Metrics suite and accurate for prediction of QoS parameters than the LSSVM (HMS), Object-Oriented source code metrics (OOM), metrics method with linear and polynomial kernel functions. Further- extracted using Principal Component Analysis (PCA) method more, we also observe that the predictive model created using object-oriented metrics achieves better results in comparison to and metrics selected using Rough Set Analysis (RSA) are other sets of source code metrics. considered as input to develop a QoS prediction model. The Index Terms—LSSVM, Machine Learning, Service Oriented study presented in this paper is an extension of our previous Computing, Source Code Metrics, Web Services, Quality of work on predicting QoS parameters using Extreme Learning Service (QoS) Machines [7]. While ELM has been used in the past for QoS parameter prediction, the application of LSSVM is novel and I. I NTRODUCTION unique in context to exiting work. This research contributions Service Oriented Computing and Architecture (SOA) of the study presented in this paper are the following: paradigm consists of assembling and combining loosely cou- 1) Application of 37 source-code metrics for prediction pled software components called as services for developing of 15 different Web Service QoS parameters by using distributed system. Prediction of Web Service QoS parameters LSSVM machine learning classifier with three different is important for both the developers and consumers of the variants of kernel functions. service [7]. One of the major objectives of a Web Service 2) Application of two feature selection techniques i.e., PCA provider is the ability to estimate and subsequently improve and RSA to select suitable set of source code metrics the QoS parameters associated with the given Web Services. for building a predictive model. One of the approaches for estimating and improving the QoS parameters is to compute source-code metrics during II. R ELATED W ORK the development phase. Predicting quality of Object-Oriented Several researchers have investigated the impact of Object- (OO) Software System using different kinds of source code Oriented (OO) source code metrics on software quality and metrics is an area which has attracted several researchers’ observed that OO metrics have a strong influence on software attention in the past [2][22][11][5]. However, predicting QoS quality attributes. Research shows that the quality of OO parameters for Web Services using source code metrics is software can be estimated using several source code metrics a relatively unexplored area. In the study presented in this [5] [10] [3] [9][8]. Bingu Shim et al. have defined five dif- paper, we conduct an experiment on fifteen different quality of ferent quality parameters i.e., effectiveness, flexibility, discov- service parameters such as Availability, Best Practices, Com- erability, reusability and understandability for service oriented pliance, Conformity, Documentation, Interoperability, Latency, applications [16]. Mikhail et al. have defined SCMs in order to Maintainability, Modularity, Response Time, Reusability, Reli- measure the structural coupling & cohesion of service-oriented ability, Successability, Throughput, and Testability, using thirty systems [14][15]. Vuong Xuan Tran et al. proposed a novel seven different source code metrics on a dataset consisting approach to design and develop QoS systems and describe of two hundred real-world Web Services. We compute thirty an algorithm to evaluate its ranking in order to compute the seven source code metrics and use them as input to develop quality of Web services [19]. Cristian Mateos et al. analyzed Copyright © 2017 for this paper by its authors. 66 5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2017) the available approaches to remove undesirable anti-patterns LSSVM based approach. The Web Service dataset provides the using code-first [12]. Ping Wang proposed another decision quality of service parameters values such as response time, model under obscure data to choose a Web Service [21]. availability, throughput, compliance, latency for 2507 Web Services. The QoS parameters values by the dataset provider III. R ESEARCH F RAMEWORK are computed using Web service benchmark tools [1]. In this A. Dependent Variables- QoS Parameters study, we use 200 Web Services for the analysis. The reason Al-Masri et al. define 9 quality of service parameters of for selection of 200 web-services is stated in our earlier work Web Services. They compute the QoS parameters using Web [7] as the study presented in this paper is an extension of the service benchmark tools. The QoS parameters are: Availability previous work. (AV), Best Practices (BP), Compliance (CP), Documentation D. Feature Extraction using Principal Component Analysis (DOC), Latency (LT), Response Time (RT), Reliability (REL), (PCA) Successability (SA), Throughput (TP), Maintainability, Modu- larity, Reusability, Testability, Interoperability and Conformity. We perform feature extraction using Principal Component These QoS parameters are the dependent variables for the Analysis (PCA). The main motivation of using PCA is for machine learning model. transforming high dimension data space into lower dimension data space. The lower dimension data consists of the most B. Predictor Variables: Source Code Metrics significant features [20]. We label the new metrics (or features) Three different types of metrics suite are considered as after applying PCA as principal component domain metrics. independent or predictor variables Figure 1 displays the steps followed by us to extract the feature 1) Object-Oriented Source Code Metrics: We compute set using PCA. nineteen different Object-Oriented source code metrics from the bytecode of the compiled Java files of the Web Services Eigen value and Eigen in our experimental dataset using CKJM extended tool1 [5]. Feature vector computation using MATLAB normalization - CKJM extended is an extended version of tool for calculating zero mean value command Data Set (eign = eig(data)) Chidamber and Kemerer Java source code metrics and many other metrics such as average method complexity, McCabe’s Cyclomatic Complexity, lack of cohesion among the classes. Java class files from the WSDL file are generated using Principal components Reduced set of WSDL2Java Axis2 code generator2 , which is available as an features (metrics) selection based on eigenvalue being greater Eclipse plug-in. We then compiled the Java files to generate are evaluated than 1.0 the bytecode for computing the size and structure of software metrics using the CKJM extended tool. Fig. 1: Sequence of Steps for Applying PCA 2) Henry M. Sneed WSDL Metric Suite: Sneed et al. develop a tool for measuring Web Service interfaces [17][6]. We apply PCA with varimax rotation technique on all the The suite primarily consists of six different source code software metrics. Table I shows the result and outcome of PCA metrics to measure complexity of service interfaces: Data with varimax rotation method. Table I reveals the relationship Flow Complexity, Interface Relation Complexity, Interface between domain metrics and original software metrics. For Data Complexity, Interface Structure Complexity, Interface each principal component (PC), % variance, % cumulative and Format Complexity and Language Complexity. These metrics interpreted metrics set are presented in Table I. are statically computed from a service interface in WSDL. The TABLE I: Principle Component Analysis (PCA) Results metrics are based on analyzing the WSDL schema elements PC Eigenvalue variance % % Cumulative Interpreted Metrics [17][6]. PC1 6.40 17.30 17.30 Ce, Ca, RFC, CBO, LCO, LCOM3, CAM, DAM 3) Baski and Misra Metrics: Baski and Misra proposed a PC2 5.8 15.76 33.06 DP, FP, OP, MRS, OPS, IDFC, IRC tool to compute six different complexity metrics of WSDL file PC3 3.67 9.94 43.00 CE, MiRV, MDC, MeRV, DW, MR PC4 3.39 9.16 52.17 ILC, DMR, ISC, IDC [4]. These metrics are based on the analysis of the structure PC5 3.34 9.03 61.2 MOA, CBM, IC PC6 2.50 6.77 67.98 MFA, NOC, DIT, IFC of the exchanged messages described in WSDL file which PC7 2.23 6.02 74.00 NPM, WMC becomes the basis for computing the data complexity. These PC8 2.14 5.79 79.79 AMC, MRV PC9 1.36 3.7 83.5 LCOM metrics are based on analyzing the WSDL and XSD schema elements [4]. E. Feature Selection using Rough Set Analysis (RSA) C. Experimental Dataset Before the application of RSA, the input data need to In our study, the Web Service dataset collected by Al-Masri be categorized. In our study, K-means clustering approach et al. 3 is used to measure the performance of the proposed is applied for the purpose of data categorization. In the 1 http://gromit.iiar.pwr.wroc.pl/p inf/ckjm/ approach, the data belonging to a particular cluster are grouped 2 https://sourceforge.net/projects/wsdl2javawizard/ under a single category or class. After the application of K- 3 http://www.uoguelph.ca/∼qmahmoud/qws/ means clustering approach, we obtain 3 clusters and the data Copyright © 2017 for this paper by its authors. 67 5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2017) were categorized into three groups: High, Medium, and Low Availability correlation. Figure 1 shows the steps followed to identify the best set (in-terms of relevance) of features using RSA. AM Response Time Feature Successability Calculation of Lower Categorization into and upper groups using approximation of all k-means clustering BMS Throughput Data Set method possible sets Compliance Identification of Computation of Reliability HMS best set of Accuracy for all source-code metrics possible sets Latency Fig. 2: Rough Set Analysis (RSA) based Feature Selection OOM Best Practices Table II displays the selected set of source-code metrics for every QoS parameter. The resultant set is derived from the Maintainability original set of 37 source-code metrics using RSA approach. PCA Documentation TABLE II: Source Code Metrics Identified using Rough Set RSA Reusability Analysis (RSA) QoS Selected Metrics Modularity Availability MiRV, Ca, CC, CAM, IC, MFA, LC, SC, LCOM3, WMC, FC, MeRV Response Time Ca, DMR, LC, SC, WMC, MFA, CC, IC, CAM, LCOM3 Successability CAM, LCOM3, DAM, FC, LC, DFC, MRV, ME, SC, WMC, LCO, MOA Independent Variables Throughput MiRV, Ce, CC, CAM, ME, MFA, LC, SC, CBM, MRV, FC, MeRV, MOA Interoperability Compliance MiRV, NPM, CC, WMC, CAM, MOA, SC, LC, FC, DFC, ME, Ca, MRV, DAM Reliability LCOM3, MFA, FC, LC, DFC, CAM, SC, WMC, LCO, MOA Latency IC, FC, LC, DMR, MRV, MOA, ME, CAM, DC, DFC, NOC, LCO, NPM Best Practices MiRV, Ca, CC, CAM, ME, MFA, LC, SC, MRV, FC, MOA, WMC, DFC, NPM Testability Maintainability CBM, DP, LCOM3, MFA, Ce, CAM, MOA Documentation CC, LC, ME, IC, SC, CAM, Ca, DFC, MRV, WMC, MeRV, FC, NPM Reusability LCOM3, FC, MDC, LCOM, DMR, SC, LC, DFC Modularity AMC, LC, DMR, SC, Ca, IC, DFC, ME, DP, MiRV, MOA, MRV, WMC Conformity Interoperability MiRV, CC, SC, MeRV, LC, WMC, MFA, DIT, CBO Testability FC, CC, RFC, ME, NOC, MiRV, DIT, SC, LC Conformity CAM, Ca, ME, DFC, FC, WMC, MRV, LC Dependent Variables Fig. 3: Dependent and Independent Variables F. Effectiveness of Metrics Once we have the QoS data, the relationship or degree of association between source code metrics and QoS parame- generate models for predicting QoS parameters. The different ters can be determined. The set of source code metrics are set of source code metrics are used a input of the models. We considered as independent variables and QoS parameters are also examine LSSVM different kernel functions to investigate considered as a dependent variables. In our experiments, six if we can achieve better result and compare the performance different set of source metrics (all metrics (AM), Baski and of various kernel functions. Misra metrics suite, Henry M. Sneed WSDL metric suite, The block diagram displayed in Figure 4, illustrates the object-oriented metrics suite, selected set of metrics princi- sequence of steps used to determine the predicted quality of pal component analysis (PCA), and selected set of metrics service (QoS) parameters using the LSSVM method with three using rough set analysis (RSA) are considered as input to different types of kernel functions. In our work, the following develop fifteen QoS parameters (Response Time Availabil- steps are performed to generate quality of service prediction ity, Throughput, Successability, Reliability, Compliance, Best models: Practices, Latency, Documentation, Maintainability, Modular- 1) Source code metrics computation for all the Web Ser- ity, Reusability, Testability, Interoperability, and Conformity) vices in the data-set as described in Section III-B. prediction models. Figure 3 shows the independent and de- 2) Selection of suitable set of source code metrics using pendent variables used for QoS parameter prediction model. PCA and RSA feature extraction and selection tech- From Figure 3, we infer that a total of eight different sets of niques. independent variables are possible for each QoS parameter. 3) Predictive model generation by considering source code metrics as input to estimate fifteen different quality of IV. P ROPOSED M ACHINE L EARNING BASED A PPROACH service parameters. Least Square Support Vector Machines (LSSVM) are super- 4) Identification of performance measures to evaluate the vised learning methods having wide range of applications doe predictive ability and effectiveness of quality of service classification, regression and outliers detection problems [18]. prediction models. In our experiments, we use LSSVM as regression technique to 5) Application of validation methods to determine the true Copyright © 2017 for this paper by its authors. 68 5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2017) Baski and Misra Performance Metrics (BMS) Linear Kerenl Evaluation Baski and Baski and Misra Metric Misra Metrics suite Henry M. Sneed WSDL File Metrics (HMS) Henry M. Sneed Tool Polynomial Sneed Metrics Statistical Test Object Kerenl Oriented QoS metrics WSDL to Java (OOM) Object- All metrics (AM) CKJM Source Code Oriented Model Extended RBF Kerenl Metrics PCA comparsion RSA Set of source code metrics Least Square Support Vector Validation of developed models Machines (LSSVM method) Fig. 4: Proposed Steps Used for the QoS Prediction (A Multi-Step Process) predictive applicability of the predictive models. D. Performance Parameter 6) Application of rigorous statistical significance tests to In order to evaluate the QoS prediction model, various compare the performance of one prediction technique performance parameters are defined in the machine learning over other approaches and also determine the superiority literature to measure the effectiveness of the QoS prediction of one set of source code metrics over the other sets. models. In this work, we consider three different performance parameters: Mean Magnitude Relative Error (MMRE), Mean A. Computation of Source Code Metrics Absolute Error (MAE), and Root Mean Square Error (RMSE) During our experimental analysis, we consider 200 Web Ser- to evaluate the QoS prediction model [13]. A lower value for vices as the experimental dataset (same as used by Kumar et these performance parameters denotes an effective prediction al. [7]). In this work, WSDL interface complexity metrics and model. WSDL complexity metrics are computed using Henry Sneed metrics tool and Baski and Misra metrics tool respectively. E. Validation Method Then we use the wsimport 4 tool to parse WSDL document file The objective of the study presented in this paper is to of a Web Service and generate its corresponding Java class. build and apply statistical models to predict different QoS This involves extracting the Java source code implementing parameters for future releases and unseen similar natured the service. As shown in Figure 4, we compiled the Java files projects. Hence, it is necessary to validate the developed QoS to generate the bytecode for computing the size and structure model on a different data-sets than the dataset on which the software metrics using the CKJM extended tool. training is done. In our experiments, we consider the standard k-fold cross validation approach (we take k = 10) to validate B. Feature Selection Method the proposed QoS model. In our analysis, we also perform In our experiments, we examine 37 different software outlier detection analysis to eliminate the extreme values metrics (Chidamber and Kemerer, Harry M. Sneed, Baski which may add to the noise & bias in the model performance & Misra)) to predict 15 different QoS attributes. It is very and accuracy results. The outlier analysis is doen based on the essential to remove irrelevant and unimportant source code following equations: metrics out of these source code metrics so that only relevant  source code metrics are included in the construction of QoS if |yji − yˆj | > 3 ∗ σ for Effective outliers ei = prediction models. In order to achieve the stated objective, we if |yji − yˆj | <= 3 ∗ σ for Non Effective outliers consider two different features selection techniques: principal (1) component analysis and rough set analysis for feature selec- F. Statistical Significance Tests and Procedures tion. In order to bring rigour and mitigate threats to validity in C. Prediction Techniques our analysis of the results, we apply pairwise t-test approach. We conduct t-test to determine which prediction method and In this work, we have use LSSVM with three different feature selection techniques performs relatively better or does kernel functions to develop QoS prediction model. the models perform equally well. We analyze all the results 4 http://docs.oracle.com/javase/6/docs/technotes/ based on the 0.05 significance level, i.e. two models are tools/share/wsimport.html significantly different (null hypothesis rejected) if the p-value Copyright © 2017 for this paper by its authors. 69 5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2017) is less than 0.05 (the cut-off valie) else there is no significant Table III displays the MMRE, MAE, and RMSE values different between them (null hypothesis accepted). obtained after applying LSSVM method with linear kernel, polynomial kernel and RBF kernel functions. Table III pro- 0.4 vides complete and detailed results for all the QoS parameters, feature extraction and selecion techniques, metrics suites and 0.3 performance evaluation methods. From Table III, we make the 0.2 following observations: Pearson residual 0.1 0 1) In case of linear kernel function, we observe that the -0.1 model built by considering selected set of metrics using -0.2 RSA as input has low values of MMRE, MAE and -0.3 RMSE in comparison with other sets of metrics. This -0.4 clearly implies that the performance of the model de- -0.5 veloped using RSA is much better than the performance BMS HMS OOM AM PCA RSA of other models i.e., low vales of MMRE, MAE, and Fig. 5: Box-Plot Visual Analysis (Linear Kernel) RMSE for QoS prediction as compared to a model developed using other sets of metrics. 2) In case of polynomial kernel function, we observe that the model built by considering all metrics has low value of MMRE, MAE and RMSE in comparison to other 0.3 sets of metrics. We thus infer that the performance of 0.2 the prediction model developed using all metrics (AM) is Pearson residual 0.1 much better than other models i.e., low vales of MMRE, 0 MAE, and RMSE for QoS prediction as compared to a -0.1 model developed using other sets of metrics. -0.2 3) In case of RBF kernel function, we notice that the model -0.3 developed by considering Baski and Misra Metric has -0.4 low value of MMRE, MAE, and RMSE in comparison with other sets of metrics. This implies that the per- BMS HMS OOM AM PCA RSA formance of the model developed using BMS is much Fig. 6: Box-Plot Visual Analysis (Polynomial Kernel) better than other models i.e., low vales of MMRE, MAE, and RMSE for QoS prediction as compared to a model developed using other sets of metrics. 0.8 In order to have a visual comparion (visual analytics) of the results, we draw the Pearson residual boxplots of models 0.6 developed using LSSVM method with linear kernel function. Pearson residual 0.4 The plots are displayed in Figure 5. The middle line of each 0.2 box in Figure 5 shows the median value of Pearson residual. 0 From Figure 5, we observe that all models built have a median -0.2 residual value close to zero and the model developed by -0.4 considering selected set of metrics using RSA has smallest -0.6 whiskers, narrowest box as well as few numbers of outliers. This shows that model developed by considering selected set BMS HMS OOM AM PCA RSA of metrics using RSA as input results in better performance as compared to other metrics. Figure 6 shows the Pearson Fig. 7: Box-Plot Visual Analysis (RBF Kernel) residual boxplots of models developed using LSSVM method with polynomial kernel function. From Figure 6, we infer that V. E XPERIMENTAL A NALYSIS all models have a median residual value close to zero and In this paper, LSSVM method with three different types of the model developed based on selected set of metrics using kernel functions have been considered to develop a model to AM has smallest whiskers, narrowest box as well as few predict fifteen different QoS parameters by considering six numbers of outliers. This result shows that model developed different set of source code metrics as input. The detailed by considering AM as input results in better performance as descriptions of these source code metrics were already pre- compared to others. sented in section III. The performance of the developed QoS Figure 7 shows the Pearson residual boxplots of models prediction models are compared using three different types of developed using LSSVM method with RBF kernel function. performance parameters i.e., MMRE, MAE and RMSE. From Figure 7, it is observed that all developed models have Copyright © 2017 for this paper by its authors. 70 5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2017) TABLE III: Performance Matrix for LSSVM method with Linear, Polynomial and RBF Kernel and All Different Metrics Suites ity ility ime ion ces ity il ntat t erab nce se T nab ity ility ty hpu abil lity acti ity ity i form ular plia ume ency sabi ntai abil abil t Pr ilab cess oug rop pon Com Mod Con Test Reli Mai Reu Inte Thr Doc Ava Res Suc Lat Bes Linear Kernel MMRE AM 0.32 0.32 0.35 0.74 0.60 0.68 0.56 0.46 0.32 0.90 0.29 0.14 0.16 0.13 0.06 OOM 0.31 0.33 0.34 0.73 0.59 0.69 0.51 0.36 0.39 0.89 0.46 0.44 0.42 0.49 0.25 WIM 0.32 0.34 0.35 0.74 0.61 0.67 0.56 0.38 0.49 0.92 0.42 0.22 0.20 0.18 0.06 WSM 0.31 0.33 0.35 0.75 0.60 0.68 0.56 0.39 0.72 0.91 0.66 0.41 0.35 0.43 0.21 RSA 0.31 0.34 0.34 0.65 0.60 0.69 0.56 0.34 0.84 0.94 0.62 0.32 0.38 0.30 0.10 PCA 0.31 0.34 0.35 0.74 0.61 0.66 0.55 0.37 0.54 0.92 0.61 0.33 0.31 0.33 0.13 MAE AM 0.13 0.09 0.16 0.19 0.25 0.24 0.11 0.19 0.08 0.21 0.09 0.04 0.08 0.04 0.03 OOM 0.13 0.09 0.16 0.18 0.25 0.24 0.1 0.17 0.10 0.21 0.13 0.12 0.18 0.13 0.14 WIM 0.14 0.10 0.17 0.18 0.26 0.24 0.11 0.18 0.11 0.22 0.11 0.06 0.10 0.04 0.03 WSM 0.14 0.10 0.17 0.18 0.26 0.24 0.11 0.18 0.17 0.22 0.16 0.11 0.15 0.11 0.13 RSA 0.13 0.10 0.16 0.16 0.25 0.24 0.11 0.17 0.20 0.23 0.15 0.10 0.18 0.10 0.06 PCA 0.14 0.10 0.17 0.18 0.26 0.23 0.11 0.18 0.13 0.22 0.15 0.10 0.14 0.09 0.08 RMSE AM 0.20 0.13 0.22 0.22 0.29 0.27 0.14 0.23 0.11 0.26 0.11 0.05 0.1 0.05 0.04 OOM 0.19 0.14 0.21 0.22 0.29 0.27 0.13 0.21 0.13 0.27 0.17 0.17 0.22 0.18 0.17 WIM 0.20 0.15 0.22 0.21 0.30 0.27 0.15 0.22 0.16 0.28 0.15 0.08 0.13 0.07 0.05 WSM 0.20 0.15 0.22 0.21 0.29 0.27 0.15 0.22 0.22 0.28 0.21 0.15 0.20 0.16 0.16 RSA 0.19 0.15 0.22 0.2 0.29 0.27 0.14 0.2 0.25 0.29 0.20 0.15 0.23 0.14 0.10 PCA 0.20 0.15 0.22 0.21 0.30 0.26 0.14 0.21 0.17 0.28 0.19 0.14 0.18 0.13 0.11 Polynomial Kernel MMRE AM 0.32 0.33 0.35 0.74 0.61 0.68 0.56 0.48 0.73 0.90 0.70 0 0.37 0 0.25 OOM 0.31 0.33 0.34 0.73 0.59 0.70 0.55 0.38 1.02 0.89 0.61 0.59 0.51 0.53 0.37 WIM 0.32 0.34 0.35 0.74 0.61 0.67 0.56 0.38 0.5 0.92 0.62 0.11 0.19 0.19 0.54 WSM 0.31 0.33 0.34 0.73 0.60 0.67 0.56 0.39 0.70 0.91 0.71 0.54 0.32 0.48 0.15 RSA 0.3 0.34 0.34 0.74 0.61 0.69 0.56 0.36 0.85 0.94 0.62 0.47 0.47 0.32 0.60 PCA 0.31 0.34 0.35 0.74 0.61 0.68 0.55 0.38 1.03 0.92 0.65 0.45 0.36 0.38 0.14 MAE AM 0.13 0.09 0.16 0.19 0.26 0.24 0.11 0.20 0.18 0.21 0.17 0 0.16 0 0.11 OOM 0.13 0.09 0.16 0.18 0.25 0.24 0.11 0.18 0.24 0.22 0.15 0.16 0.23 0.14 0.20 WIM 0.14 0.10 0.17 0.18 0.26 0.24 0.11 0.18 0.12 0.22 0.15 0.03 0.09 0.05 0.24 WSM 0.14 0.10 0.17 0.18 0.26 0.24 0.11 0.18 0.16 0.22 0.16 0.14 0.13 0.12 0.09 RSA 0.13 0.10 0.16 0.18 0.26 0.24 0.11 0.18 0.21 0.23 0.15 0.14 0.22 0.10 0.27 PCA 0.14 0.10 0.17 0.18 0.26 0.24 0.11 0.18 0.25 0.22 0.15 0.13 0.16 0.11 0.08 RMSE AM 0.20 0.14 0.22 0.22 0.29 0.28 0.14 0.23 0.22 0.26 0.22 0 0.20 0 0.14 OOM 0.19 0.14 0.21 0.22 0.29 0.28 0.14 0.22 0.30 0.27 0.20 0.22 0.27 0.20 0.22 WIM 0.20 0.15 0.22 0.21 0.30 0.27 0.15 0.22 0.16 0.28 0.19 0.05 0.12 0.08 0.25 WSM 0.20 0.14 0.22 0.21 0.29 0.27 0.15 0.22 0.22 0.28 0.22 0.19 0.18 0.18 0.12 RSA 0.19 0.15 0.22 0.21 0.30 0.28 0.14 0.21 0.26 0.29 0.20 0.19 0.26 0.14 0.29 PCA 0.20 0.15 0.22 0.21 0.30 0.27 0.14 0.21 0.30 0.28 0.20 0.17 0.19 0.15 0.1 RBF Kernel AM 0.32 0.32 0.35 0.71 0 0.67 0.38 0.45 0.09 0.38 0.08 0.03 0.08 0.05 0.02 OOM 0.3 0.32 0.34 0.57 0.13 0.68 0.28 0.36 0.22 0.01 0.40 0.42 0.20 0.44 0.13 WIM 0.32 0.34 0.35 0.27 0.42 0.39 0.45 0.29 0.08 0.92 0.16 0.03 0.07 0.02 0.02 WSM 0.3 0.33 0.33 0.50 0.60 0.67 0.56 0.37 0.39 0.89 0.30 0.35 0.16 0.24 0.05 RSA 0.3 0.34 0.34 0.60 0 0.68 0.28 0.35 0.58 0.34 0.10 0 0.16 0.13 0.09 PCA 0.31 0.33 0.35 0.73 0 0.67 0.49 0.36 0.17 0.29 0.27 0.18 0.21 0.22 0.05 MAE AM 0.13 0.09 0.16 0.18 0 0.23 0.07 0.18 0.03 0.09 0.03 0.01 0.04 0.02 0.01 OOM 0.13 0.09 0.16 0.14 0.06 0.24 0.05 0.17 0.06 0 0.11 0.12 0.09 0.11 0.07 WIM 0.14 0.10 0.17 0.07 0.18 0.14 0.09 0.14 0.03 0.22 0.04 0.01 0.04 0.01 0.01 WSM 0.13 0.10 0.16 0.12 0.26 0.23 0.11 0.18 0.10 0.22 0.07 0.09 0.06 0.06 0.02 RSA 0.13 0.10 0.16 0.15 0 0.24 0.06 0.17 0.14 0.08 0.03 0 0.07 0.04 0.05 PCA 0.14 0.10 0.17 0.18 0 0.23 0.09 0.17 0.05 0.07 0.07 0.05 0.09 0.06 0.03 RMSE AM 0.20 0.14 0.22 0.21 0 0.27 0.10 0.22 0.04 0.11 0.04 0.01 0.05 0.02 0.02 OOM 0.19 0.14 0.21 0.17 0.07 0.27 0.07 0.21 0.08 0.01 0.15 0.17 0.12 0.16 0.10 WIM 0.20 0.15 0.22 0.08 0.20 0.16 0.12 0.17 0.04 0.28 0.06 0.02 0.05 0.01 0.02 WSM 0.19 0.15 0.21 0.15 0.29 0.27 0.15 0.21 0.14 0.27 0.10 0.14 0.10 0.09 0.04 RSA 0.19 0.15 0.22 0.18 0 0.27 0.07 0.20 0.19 0.11 0.03 0 0.09 0.07 0.08 PCA 0.20 0.15 0.22 0.21 0 0.27 0.13 0.20 0.06 0.09 0.10 0.08 0.12 0.09 0.04 Copyright © 2017 for this paper by its authors. 71 5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2017) TABLE IV: Result of t-test: Among Different Metrics Set P-Value MMRE MAE RMSE AM OOM HMS BMS RSA PCA AM OOM HMS BMS RSA PCA AM OOM HMS BMS RSA PCA AM 1.000 0.063 0.031 0.063 0.031 0.031 1.000 0.031 0.031 0.031 0.031 0.031 1.000 0.031 0.031 0.031 0.031 0.031 OOM 0.063 1.000 0.031 0.031 0.031 0.031 0.031 1.000 0.031 0.031 0.031 0.031 0.031 1.000 0.031 0.031 0.031 0.031 HMS 0.031 0.031 1.000 0.031 0.094 0.031 0.031 0.031 1.000 0.063 0.031 0.031 0.031 0.031 1.000 0.313 0.031 0.031 BMS 0.063 0.031 0.031 1.000 0.031 0.031 0.031 0.031 0.063 1.000 0.031 0.031 0.031 0.031 0.313 1.000 0.031 0.031 RSA 0.031 0.031 0.094 0.031 1.000 0.031 0.031 0.031 0.031 0.031 1.000 0.031 0.031 0.031 0.031 0.031 1.000 0.031 PCA 0.031 0.031 0.031 0.031 0.031 1.000 0.031 0.031 0.031 0.031 0.031 1.000 0.031 0.031 0.031 0.031 0.031 1.000 Mean Difference MMRE MAE RMSE AM OOM HMS BMS RSA PCA AM OOM HMS BMS RSA PCA AM OOM HMS BMS RSA PCA AM 0.000 0.020 -0.392 -0.013 -0.345 -0.268 0.000 0.038 -0.082 -0.068 -0.142 -0.158 0.000 -0.052 -0.067 -0.073 -0.123 -0.148 OOM -0.020 0.000 -0.412 -0.033 -0.365 -0.288 -0.038 0.000 -0.043 -0.030 -0.103 -0.120 0.052 0.000 -0.015 -0.022 -0.072 -0.097 HMS 0.392 0.412 0.000 0.378 0.047 0.123 0.082 0.043 0.000 0.013 -0.060 -0.077 0.067 0.015 0.000 -0.007 -0.057 -0.082 BMS 0.013 0.033 -0.378 0.000 -0.332 -0.255 0.068 0.030 -0.013 0.000 -0.073 -0.090 0.073 0.022 0.007 0.000 -0.050 -0.075 RSA 0.345 0.365 -0.047 0.332 0.000 0.077 0.142 0.103 0.060 0.073 0.000 -0.017 0.123 0.072 0.057 0.050 0.000 -0.025 PCA 0.268 0.288 -0.123 0.255 -0.077 0.000 0.158 0.120 0.077 0.090 0.017 0.000 0.148 0.097 0.082 0.075 0.025 0.000 TABLE V: t-test: Among different Kernel Hence, null hypothesis is rejected only if the p-value is less P-Value than 0.05 3 = 0.0167. The null hypothesis while applying the MMRE MAE RMSE Lin Poly RBF Lin Poly RBF Lin Poly RBF Wilcoxon test is that there is no significant difference between Lin 1.000 0.000 0.000 1.000 0.000 0.000 1.000 0.000 0.000 the two classification techniques. From the result Table V, we Poly 0.000 1.000 0.000 0.000 1.000 0.000 0.000 1.000 0.000 RBF 0.000 1.000 0.000 0.000 1.000 0.000 0.000 1.000 0.000 observe that there is a significant difference between the Mean Difference kernel functions. This interpretation is due to the fact that MMRE MAE RMSE Lin Poly RBF Lin Poly RBF Lin Poly RBF the p-value is lower than 0.0167 (rejecting the null hypothesis Lin 0.000 -0.051 0.155 0.000 -0.015 0.047 0.000 -0.016 0.057 and accpeting the alternate hypothesis). However by closely Poly 0.051 0.000 0.206 0.015 0.000 0.063 0.016 0.000 0.074 RBF -0.155 -0.206 0.000 -0.047 -0.063 0.000 -0.057 -0.074 0.000 examining the value of mean difference, RBF kernel function yields better result as compared to other kernel functions. a median residual value being close to zero and the model B. Source Code Metrics Sets developed by considering selected set of metrics using BMS has smallest whiskers, narrowest box as well as few numbers In this work, six different set of source code metrics are of outliers. This shows that model developed by considering used as input to develop QoS prediction models. Hence for BMS as input obtained better performance as compared to each set of source code metrics, a total of three sets (one for others. each performance parameter) are used. Each set has 45 data points (15 QoS parameters * 3 kernel functions). Results of VI. COMPARISON OF VARIOUS K ERNEL F UNCTIONS AND Wilcoxon test with Bonferroni correction for all performance M ETRIC S UITES parameter is shown in Table V. The Bonferroni correction sets We apply Wilcoxon signed rank test to compare the per- the significance cutoff at αn , where n is number of different formance of the models using LSSVM method with three pairs (here 6 sets of metrics; n=6technique C2 = 6 ∗ 5/2 = 15) different types of kernel functions and different sets of source and all results are analyzed at a 0.05 significance level. Hence, code metrics. We use Wilcoxon test with Bonferroni correction null hypothesis is rejected only if the p-value is less than 0.05 for comparative analysis. 15 = 0.0033. From Table IV, we infer that there is no significant difference between sets of metrics. We arrive A. Kernel Functions at this conclusion due to the fact that the p-value is greater Three different types of kernel functions have been applied than 0.0033 (accepting the null hypothesis). However, by to develop QoS prediction models. Hence for each of the closely examining the value of mean difference, we infer that kernel functions, a total number of three sets (one for each per- the object-oriented Metrics are yields better performance formance parameter) are used. Each set has 90 data points (15 results in comparison to other sets of metrics. QoS parameters * 6 sets of metrics). The results of Wilcoxon test with Bonferroni correction for all performance parameters VII. T HREATS TO VALIDITY are shown in Table V. The upper portion of the Table V shows the p-value between kernel functions and the lower portion One threat to validity is that the impact on the dependent shows the mean difference value of performance parameters variable may not be completely attributed to the changes in between different kernels. The Bonferroni correction sets the the independent variable due to overfitting of the predictive significance cut-off at α n , where n is number of different model. Another threat to validity is that we have conducted pairs (3 kernel functions; n=3technique C2 = 3 ∗ 2/2 = 3) experiments on a limited dataset and the answer result can be and all results are analyzed at a 0.05 significance level. biased to the specific dataset. Copyright © 2017 for this paper by its authors. 72 5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2017) VIII. C ONCLUSION to predict maintainability of service oriented software. Our main conclusions and takeaways are the following: In High Assurance Systems Engineering (HASE), 2017 • Based on our correlation analysis of the metrics, we IEEE 18th International Symposium on, pages 88–95. conclude that there exists a high correlation between IEEE, 2017. Object-Oriented metrics and WSDL metrics. [9] L. Kumar, S. K. Rath, and A. Sureka. Using source code • From the result of Wilcoxon test with Bonferroni correc- metrics to predict change-prone web services: A case- tion, we conclude that there is a statistically significant study on ebay services. In Machine Learning Techniques difference between the performance of the predictive for Software Quality Evaluation (MaLTeSQuE), IEEE models built using three different LSSVM kernel func- Workshop on, pages 1–7. IEEE, 2017. tions. [10] W. Li and S. Henry. Maintenance metrics for the Object- • From the rest of Wilcoxon test with Bonferroni correc- Oriented paradigm. In International Software Metrics tion, we conclude that there is no statistically significant Symposium, pages 52–60, 1993. difference between different sets of source code metrics. [11] R. Malhotra and Y. Singh. On the applicability of • From Table III, we conclude that no one set of source- machine learning techniques for object oriented software code metrics dominate the other sets for any QoS param- fault prediction. Software Engineering: An International eter and vice-versa. Journal. • By assessing the value of mean difference shown in Table [12] C. Mateos, M. Crasso, A. Zunino, and J. L. O. Cos- V, we conclude that that the RBF kernel for LSSVM cia. Detecting wsdl bad practices in code–first web method yields better performance results compared to services. International Journal of Web and Grid Services, other kernel functions. 7(4):357–387, 2011. • By assessing the value of mean difference shown in [13] T. Menzies, Z. Chen, J. Hihn, and K. Lum. Selecting Table IV, we conclude that the object-oriented metrics best practices for effort estimation. IEEE Transactions yields better result compared to other sets of source code on Software Engineering, 32(11):883–895, 2006. metrics. [14] M. Perepletchikov, C. Ryan, and K. Frampton. Cohesion • Our analysis shows evidences that it is possible to esti- metrics for predicting maintainability of service-oriented mate the QoS parameters of Web Services using source software. In QSIC, pages 328–335. IEEE, 2007. code metrics and LSSVM based method. [15] M. Perepletchikov, C. Ryan, K. Frampton, and Z. Tari. Coupling metrics for predicting maintainability in R EFERENCES service-oriented designs. In ASWEC, pages 329–340. [1] E. Al-Masri and Q. H. Mahmoud. Investigating web IEEE, 2007. services on the world wide web. In Proceedings of the [16] B. Shim, S. Choue, S. Kim, and S. Park. A design quality 17th international conference on World Wide Web, pages model for service-oriented architecture. In 2008 15th 795–804. ACM, 2008. Asia-Pacific Software Engineering Conference, pages [2] V. R. Basili, L. C. Briand, and W. L. Melo. How 403–410. IEEE, 2008. reuse influences productivity in object-oriented systems. [17] H. M. Sneed. Measuring web service interfaces. In Web Communications of the ACM. Systems Evolution (WSE), pages 111–115. IEEE, 2010. [3] V. R. Basili, L. C. Briand, and W. L. Melo. A [18] J. A. Suykens, J. De Brabanter, L. Lukas, and J. Vande- validation of Object-Oriented design metrics as quality walle. Weighted least squares support vector machines: indicators. IEEE Transactions on Software Engineering, robustness and sparse approximation. Neurocomputing. 22(10):751–761, October 1996. [19] V. X. Tran, H. Tsuji, and R. Masuda. A new qos ontology [4] D. Baski and S. Misra. Metrics suite for maintainability and its qos-based ranking algorithm for web services. of extensible markup language web services. IET Soft- Simulation Modelling Practice and Theory, 17(8):1378– ware, 5(3):320–341, 2011. 1398, 2009. [5] S. R. Chidamber and C. F. Kemerer. A metrics suite for [20] D. Wang and J. Romagnoli. Robust multi-scale prin- Object-Oriented design. IEEE Transactions on Software cipal components analysis with applications to process Engineering, 20(6):476–493, June 1994. monitoring. Journal of Process Control, 15(8):869–882, [6] J. L. O. Coscia, M. Crasso, C. Mateos, and A. Zunino. 2005. Estimating web service interface quality through conven- [21] P. Wang. Qos-aware web services selection with in- tional object-oriented metrics. CLEI Electron. J, 16(1), tuitionistic fuzzy set under consumer’s vague percep- 2013. tion. Expert Systems with Applications, 36(3):4460– [7] L. Kumar, S. K. Rath, and A. Sureka. Predicting qual- 4466, 2009. ity of service (qos) parameters using extreme learning [22] Y. Zhou and H. Leung. Empirical analysis of object- machines with various kernel methods. In Quantitative oriented design metrics for predicting high and low Approaches to Software Quality, page 11, 2016. severity faults. 32(10):771–789, 2006. [8] L. Kumar, S. K. Rath, and A. Sureka. Using source code metrics and multivariate adaptive regression splines Copyright © 2017 for this paper by its authors. 73