-We conduct an empirical analysis to investigate the relationship between thirty seven different source code metrics with fifteen different Web Service QoS (Quality of Service) parameters. The source code metrics used in our experiments consists of nineteen Object-Oriented metrics, six Baski and Misra metrics, and twelve Harry M. Sneed metrics. We apply Principal Component Analysis (PCA) and Rough Set Analysis for feature extraction and selection. The different sets of metrics are provided as input to the predictive model generated using Least Square Support Vector Machine (LSSVM) with three different types of kernel functions: RBF, Polynomial, and Linear. Our experimental results reveal that the prediction model developed using LSSVM method with RBF kernel function is more effective and accurate for prediction of QoS parameters than the LSSVM method with linear and polynomial kernel functions. Furthermore, we also observe that the predictive model created using object-oriented metrics achieves better results in comparison to other sets of source code metrics. Index Terms-LSSVM, Machine Learning, Service Oriented Computing, Source Code Metrics, Web Services, Quality of Service (QoS)
Service Oriented Computing and Architecture (SOA)
paradigm consists of assembling and combining loosely
coupled software components called as services for developing
distributed system. Prediction of Web Service QoS parameters
is important for both the developers and consumers of the
service [7]. One of the major objectives of a Web Service
provider is the ability to estimate and subsequently improve
the QoS parameters associated with the given Web Services.
One of the approaches for estimating and improving the
QoS parameters is to compute source-code metrics during
the development phase. Predicting quality of Object-Oriented
(OO) Software System using different kinds of source code
metrics is an area which has attracted several researchers’
attention in the past [2][
The overall effectiveness and performance of the QoS parameter prediction models depends on the subset of source code metrics used as input to develop the statistical models. In our work, six different sets of source code metrics: all metrics (AM) for source code (thirty seven metrics), Baski and Misra Metrics suite (BMS), Harry M. Sneed Metrics suite (HMS), Object-Oriented source code metrics (OOM), metrics extracted using Principal Component Analysis (PCA) method and metrics selected using Rough Set Analysis (RSA) are considered as input to develop a QoS prediction model. The study presented in this paper is an extension of our previous work on predicting QoS parameters using Extreme Learning Machines [7]. While ELM has been used in the past for QoS parameter prediction, the application of LSSVM is novel and unique in context to exiting work. This research contributions of the study presented in this paper are the following: 1) Application of 37 source-code metrics for prediction of 15 different Web Service QoS parameters by using LSSVM machine learning classifier with three different variants of kernel functions. 2) Application of two feature selection techniques i.e., PCA and RSA to select suitable set of source code metrics for building a predictive model.
Several researchers have investigated the impact of
ObjectOriented (OO) source code metrics on software quality and
observed that OO metrics have a strong influence on software
quality attributes. Research shows that the quality of OO
software can be estimated using several source code metrics
[
Al-Masri et al. define 9 quality of service parameters of Web Services. They compute the QoS parameters using Web service benchmark tools. The QoS parameters are: Availability (AV), Best Practices (BP), Compliance (CP), Documentation (DOC), Latency (LT), Response Time (RT), Reliability (REL), Successability (SA), Throughput (TP), Maintainability, Modularity, Reusability, Testability, Interoperability and Conformity. These QoS parameters are the dependent variables for the machine learning model.
Three different types of metrics suite are considered as independent or predictor variables
nineteen different Object-Oriented source code metrics from
the bytecode of the compiled Java files of the Web Services
in our experimental dataset using CKJM extended tool1 [
develop a tool for measuring Web Service interfaces [
3) Baski and Misra Metrics: Baski and Misra proposed a tool to compute six different complexity metrics of WSDL file [4]. These metrics are based on the analysis of the structure of the exchanged messages described in WSDL file which becomes the basis for computing the data complexity. These metrics are based on analyzing the WSDL and XSD schema elements [4].
In our study, the Web Service dataset collected by Al-Masri et al. 3 is used to measure the performance of the proposed
LSSVM based approach. The Web Service dataset provides the quality of service parameters values such as response time, availability, throughput, compliance, latency for 2507 Web Services. The QoS parameters values by the dataset provider are computed using Web service benchmark tools [1]. In this study, we use 200 Web Services for the analysis. The reason for selection of 200 web-services is stated in our earlier work [7] as the study presented in this paper is an extension of the previous work.
We perform feature extraction using Principal Component Analysis (PCA). The main motivation of using PCA is for transforming high dimension data space into lower dimension data space. The lower dimension data consists of the most significant features [20]. We label the new metrics (or features) after applying PCA as principal component domain metrics. Figure 1 displays the steps followed by us to extract the feature set using PCA.
Feature normalization zero mean value Reduced set of features (metrics) are evaluated
Eigen value and Eigen vector computation using MATLAB
command (eign = eig(data)) Principal components selection based on eigenvalue being greater than 1.0
We apply PCA with varimax rotation technique on all the software metrics. Table I shows the result and outcome of PCA with varimax rotation method. Table I reveals the relationship between domain metrics and original software metrics. For each principal component (PC), % variance, % cumulative and interpreted metrics set are presented in Table I.
Before the application of RSA, the input data need to be categorized. In our study, K-means clustering approach is applied for the purpose of data categorization. In the approach, the data belonging to a particular cluster are grouped under a single category or class. After the application of Kmeans clustering approach, we obtain 3 clusters and the data were categorized into three groups: High, Medium, and Low correlation. Figure 1 shows the steps followed to identify the best set (in-terms of relevance) of features using RSA.
Once we have the QoS data, the relationship or degree of association between source code metrics and QoS parameters can be determined. The set of source code metrics are considered as independent variables and QoS parameters are considered as a dependent variables. In our experiments, six different set of source metrics (all metrics (AM), Baski and Misra metrics suite, Henry M. Sneed WSDL metric suite, object-oriented metrics suite, selected set of metrics principal component analysis (PCA), and selected set of metrics using rough set analysis (RSA) are considered as input to develop fifteen QoS parameters (Response Time Availability, Throughput, Successability, Reliability, Compliance, Best Practices, Latency, Documentation, Maintainability, Modularity, Reusability, Testability, Interoperability, and Conformity) prediction models. Figure 3 shows the independent and dependent variables used for QoS parameter prediction model. From Figure 3, we infer that a total of eight different sets of independent variables are possible for each QoS parameter.
IV. PROPOSED MACHINE LEARNING BASED APPROACH Least Square Support Vector Machines (LSSVM) are supervised learning methods having wide range of applications doe classification, regression and outliers detection problems [18]. In our experiments, we use LSSVM as regression technique to BMS HMS OOM PCA
RSA Independent Variables
Availability Response Time Successability Throughput Compliance Reliability
Latency Best Practices Maintainability Documentation
Reusability
Modularity Interoperability
Testability
Conformity Dependent Variables
Fig. 3: Dependent and Independent Variables generate models for predicting QoS parameters. The different set of source code metrics are used a input of the models. We also examine LSSVM different kernel functions to investigate if we can achieve better result and compare the performance of various kernel functions.
The block diagram displayed in Figure 4, illustrates the sequence of steps used to determine the predicted quality of service (QoS) parameters using the LSSVM method with three different types of kernel functions. In our work, the following steps are performed to generate quality of service prediction models: 1) Source code metrics computation for all the Web Services in the data-set as described in Section III-B. 2) Selection of suitable set of source code metrics using PCA and RSA feature extraction and selection techniques. 3) Predictive model generation by considering source code metrics as input to estimate fifteen different quality of service parameters. 4) Identification of performance measures to evaluate the predictive ability and effectiveness of quality of service prediction models. 5) Application of validation methods to determine the true
WSDL to Java Source Code Baski and Misra Metric
suite Sneed Tool
CKJM Extended
Baski and Misra Metrics
Henry M.
Sneed Metrics QoS ObjectOriented Metrics Baski and Misra Metrics (BMS) Henry M. Sneed Metrics (HMS)
All metrics (AM)
PCA predictive applicability of the predictive models. 6) Application of rigorous statistical significance tests to compare the performance of one prediction technique over other approaches and also determine the superiority of one set of source code metrics over the other sets.
During our experimental analysis, we consider 200 Web Services as the experimental dataset (same as used by Kumar et al. [7]). In this work, WSDL interface complexity metrics and WSDL complexity metrics are computed using Henry Sneed metrics tool and Baski and Misra metrics tool respectively. Then we use the wsimport 4 tool to parse WSDL document file of a Web Service and generate its corresponding Java class. This involves extracting the Java source code implementing the service. As shown in Figure 4, we compiled the Java files to generate the bytecode for computing the size and structure software metrics using the CKJM extended tool.
In our experiments, we examine 37 different software metrics (Chidamber and Kemerer, Harry M. Sneed, Baski & Misra)) to predict 15 different QoS attributes. It is very essential to remove irrelevant and unimportant source code metrics out of these source code metrics so that only relevant source code metrics are included in the construction of QoS prediction models. In order to achieve the stated objective, we consider two different features selection techniques: principal component analysis and rough set analysis for feature selection.
In this work, we have use LSSVM with three different kernel functions to develop QoS prediction model.
tools/share/wsimport.html
In order to evaluate the QoS prediction model, various performance parameters are defined in the machine learning literature to measure the effectiveness of the QoS prediction models. In this work, we consider three different performance parameters: Mean Magnitude Relative Error (MMRE), Mean Absolute Error (MAE), and Root Mean Square Error (RMSE) to evaluate the QoS prediction model [13]. A lower value for these performance parameters denotes an effective prediction model.
The objective of the study presented in this paper is to build and apply statistical models to predict different QoS parameters for future releases and unseen similar natured projects. Hence, it is necessary to validate the developed QoS model on a different data-sets than the dataset on which the training is done. In our experiments, we consider the standard k-fold cross validation approach (we take k = 10) to validate the proposed QoS model. In our analysis, we also perform outlier detection analysis to eliminate the extreme values which may add to the noise & bias in the model performance and accuracy results. The outlier analysis is doen based on the following equations: ei = if jyji if jyji y^j j > 3 y^j j <= 3 for Effective outliers for Non Effective outliers (1)
In order to bring rigour and mitigate threats to validity in our analysis of the results, we apply pairwise t-test approach. We conduct t-test to determine which prediction method and feature selection techniques performs relatively better or does the models perform equally well. We analyze all the results based on the 0:05 significance level, i.e. two models are significantly different (null hypothesis rejected) if the p-value is less than 0:05 (the cut-off valie) else there is no significant different between them (null hypothesis accepted).
BMS
In this paper, LSSVM method with three different types of kernel functions have been considered to develop a model to predict fifteen different QoS parameters by considering six different set of source code metrics as input. The detailed descriptions of these source code metrics were already presented in section III. The performance of the developed QoS prediction models are compared using three different types of performance parameters i.e., MMRE, MAE and RMSE.
Table III displays the MMRE, MAE, and RMSE values obtained after applying LSSVM method with linear kernel, polynomial kernel and RBF kernel functions. Table III provides complete and detailed results for all the QoS parameters, feature extraction and selecion techniques, metrics suites and performance evaluation methods. From Table III, we make the following observations: 1) In case of linear kernel function, we observe that the model built by considering selected set of metrics using RSA as input has low values of MMRE, MAE and RMSE in comparison with other sets of metrics. This clearly implies that the performance of the model developed using RSA is much better than the performance of other models i.e., low vales of MMRE, MAE, and RMSE for QoS prediction as compared to a model developed using other sets of metrics. 2) In case of polynomial kernel function, we observe that the model built by considering all metrics has low value of MMRE, MAE and RMSE in comparison to other sets of metrics. We thus infer that the performance of the prediction model developed using all metrics (AM) is much better than other models i.e., low vales of MMRE, MAE, and RMSE for QoS prediction as compared to a model developed using other sets of metrics. 3) In case of RBF kernel function, we notice that the model developed by considering Baski and Misra Metric has low value of MMRE, MAE, and RMSE in comparison with other sets of metrics. This implies that the performance of the model developed using BMS is much better than other models i.e., low vales of MMRE, MAE, and RMSE for QoS prediction as compared to a model developed using other sets of metrics.
In order to have a visual comparion (visual analytics) of the results, we draw the Pearson residual boxplots of models developed using LSSVM method with linear kernel function. The plots are displayed in Figure 5. The middle line of each box in Figure 5 shows the median value of Pearson residual. From Figure 5, we observe that all models built have a median residual value close to zero and the model developed by considering selected set of metrics using RSA has smallest whiskers, narrowest box as well as few numbers of outliers. This shows that model developed by considering selected set of metrics using RSA as input results in better performance as compared to other metrics. Figure 6 shows the Pearson residual boxplots of models developed using LSSVM method with polynomial kernel function. From Figure 6, we infer that all models have a median residual value close to zero and the model developed based on selected set of metrics using AM has smallest whiskers, narrowest box as well as few numbers of outliers. This result shows that model developed by considering AM as input results in better performance as compared to others.
Figure 7 shows the Pearson residual boxplots of models developed using LSSVM method with RBF kernel function. From Figure 7, it is observed that all developed models have AM OOM WIM WSM RSA PCA AM OOM WIM WSM RSA PCA AM OOM WIM WSM RSA PCA AM OOM WIM WSM RSA PCA AM OOM WIM WSM RSA PCA AM OOM WIM WSM RSA PCA AM OOM WIM WSM RSA PCA AM OOM WIM WSM RSA PCA AM OOM WIM WSM RSA PCA 0.90 0.89 0.92 0.91 0.94 0.92 a median residual value being close to zero and the model developed by considering selected set of metrics using BMS has smallest whiskers, narrowest box as well as few numbers of outliers. This shows that model developed by considering BMS as input obtained better performance as compared to others.
VI. COMPARISON OF VARIOUS KERNEL FUNCTIONS AND
METRIC SUITES
We apply Wilcoxon signed rank test to compare the performance of the models using LSSVM method with three different types of kernel functions and different sets of source code metrics. We use Wilcoxon test with Bonferroni correction for comparative analysis.
Three different types of kernel functions have been applied to develop QoS prediction models. Hence for each of the kernel functions, a total number of three sets (one for each performance parameter) are used. Each set has 90 data points (15 QoS parameters * 6 sets of metrics). The results of Wilcoxon test with Bonferroni correction for all performance parameters are shown in Table V. The upper portion of the Table V shows the p-value between kernel functions and the lower portion shows the mean difference value of performance parameters between different kernels. The Bonferroni correction sets the significance cut-off at n , where n is number of different pairs (3 kernel functions; n=3techniqueC2 = 3 2=2 = 3) and all results are analyzed at a 0:05 significance level.
RSA 0.031 0.031 0.031 0.031 1.000 0.031 RSA -0.142 -0.103 -0.060 -0.073 0.000 0.017 Hence, null hypothesis is rejected only if the p-value is less than 0:05 = 0:0167. The null hypothesis while applying the 3 Wilcoxon test is that there is no significant difference between the two classification techniques. From the result Table V, we observe that there is a significant difference between the kernel functions. This interpretation is due to the fact that the p-value is lower than 0:0167 (rejecting the null hypothesis and accpeting the alternate hypothesis). However by closely examining the value of mean difference, RBF kernel function yields better result as compared to other kernel functions.
In this work, six different set of source code metrics are used as input to develop QoS prediction models. Hence for each set of source code metrics, a total of three sets (one for each performance parameter) are used. Each set has 45 data points (15 QoS parameters * 3 kernel functions). Results of Wilcoxon test with Bonferroni correction for all performance parameter is shown in Table V. The Bonferroni correction sets the significance cutoff at n , where n is number of different pairs (here 6 sets of metrics; n=6techniqueC2 = 6 5=2 = 15) and all results are analyzed at a 0.05 significance level. Hence, null hypothesis is rejected only if the p-value is less than 01:055 = 0:0033. From Table IV, we infer that there is no significant difference between sets of metrics. We arrive at this conclusion due to the fact that the p-value is greater than 0:0033 (accepting the null hypothesis). However, by closely examining the value of mean difference, we infer that the object-oriented Metrics are yields better performance results in comparison to other sets of metrics.
One threat to validity is that the impact on the dependent variable may not be completely attributed to the changes in the independent variable due to overfitting of the predictive model. Another threat to validity is that we have conducted experiments on a limited dataset and the answer result can be biased to the specific dataset.