=Paper=
{{Paper
|id=Vol-2017/paper11
|storemode=property
|title=Estimating Web Service Quality of Service Parameters using Source Code Metrics and LSSVM
|pdfUrl=https://ceur-ws.org/Vol-2017/paper11.pdf
|volume=Vol-2017
|authors=Lov Kumar,Santanu Rath,Ashish Sureka
|dblpUrl=https://dblp.org/rec/conf/apsec/KumarRS17
}}
==Estimating Web Service Quality of Service Parameters using Source Code Metrics and LSSVM==
https://ceur-ws.org/Vol-2017/paper11.pdf
5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2017)
Estimating Web Service Quality of Service
Parameters using Source Code Metrics and LSSVM
Lov Kumar Santanu Rath Ashish Sureka
NIT Rourkela, India NIT Rourkela, India Ashoka University, India
lovkumar505@gmail.com skrath@nitrkl.ac.in ashish.sureka@ashoka.edu.in
Abstract—We conduct an empirical analysis to investigate the a model using LSSVM method with three different types of
relationship between thirty seven different source code metrics kernel functions: linear kernel, polynomial kernel and RBF
with fifteen different Web Service QoS (Quality of Service) kernel. LSSVM method is a least square version of support
parameters. The source code metrics used in our experiments
consists of nineteen Object-Oriented metrics, six Baski and vector machine (SVM) and is based on statistical learning
Misra metrics, and twelve Harry M. Sneed metrics. We apply theory [18].
Principal Component Analysis (PCA) and Rough Set Analysis for The overall effectiveness and performance of the QoS
feature extraction and selection. The different sets of metrics are parameter prediction models depends on the subset of source
provided as input to the predictive model generated using Least code metrics used as input to develop the statistical models.
Square Support Vector Machine (LSSVM) with three different
types of kernel functions: RBF, Polynomial, and Linear. Our In our work, six different sets of source code metrics: all
experimental results reveal that the prediction model developed metrics (AM) for source code (thirty seven metrics), Baski
using LSSVM method with RBF kernel function is more effective and Misra Metrics suite (BMS), Harry M. Sneed Metrics suite
and accurate for prediction of QoS parameters than the LSSVM (HMS), Object-Oriented source code metrics (OOM), metrics
method with linear and polynomial kernel functions. Further- extracted using Principal Component Analysis (PCA) method
more, we also observe that the predictive model created using
object-oriented metrics achieves better results in comparison to and metrics selected using Rough Set Analysis (RSA) are
other sets of source code metrics. considered as input to develop a QoS prediction model. The
Index Terms—LSSVM, Machine Learning, Service Oriented study presented in this paper is an extension of our previous
Computing, Source Code Metrics, Web Services, Quality of work on predicting QoS parameters using Extreme Learning
Service (QoS) Machines [7]. While ELM has been used in the past for QoS
parameter prediction, the application of LSSVM is novel and
I. I NTRODUCTION unique in context to exiting work. This research contributions
Service Oriented Computing and Architecture (SOA) of the study presented in this paper are the following:
paradigm consists of assembling and combining loosely cou- 1) Application of 37 source-code metrics for prediction
pled software components called as services for developing of 15 different Web Service QoS parameters by using
distributed system. Prediction of Web Service QoS parameters LSSVM machine learning classifier with three different
is important for both the developers and consumers of the variants of kernel functions.
service [7]. One of the major objectives of a Web Service 2) Application of two feature selection techniques i.e., PCA
provider is the ability to estimate and subsequently improve and RSA to select suitable set of source code metrics
the QoS parameters associated with the given Web Services. for building a predictive model.
One of the approaches for estimating and improving the
QoS parameters is to compute source-code metrics during II. R ELATED W ORK
the development phase. Predicting quality of Object-Oriented Several researchers have investigated the impact of Object-
(OO) Software System using different kinds of source code Oriented (OO) source code metrics on software quality and
metrics is an area which has attracted several researchers’ observed that OO metrics have a strong influence on software
attention in the past [2][22][11][5]. However, predicting QoS quality attributes. Research shows that the quality of OO
parameters for Web Services using source code metrics is software can be estimated using several source code metrics
a relatively unexplored area. In the study presented in this [5] [10] [3] [9][8]. Bingu Shim et al. have defined five dif-
paper, we conduct an experiment on fifteen different quality of ferent quality parameters i.e., effectiveness, flexibility, discov-
service parameters such as Availability, Best Practices, Com- erability, reusability and understandability for service oriented
pliance, Conformity, Documentation, Interoperability, Latency, applications [16]. Mikhail et al. have defined SCMs in order to
Maintainability, Modularity, Response Time, Reusability, Reli- measure the structural coupling & cohesion of service-oriented
ability, Successability, Throughput, and Testability, using thirty systems [14][15]. Vuong Xuan Tran et al. proposed a novel
seven different source code metrics on a dataset consisting approach to design and develop QoS systems and describe
of two hundred real-world Web Services. We compute thirty an algorithm to evaluate its ranking in order to compute the
seven source code metrics and use them as input to develop quality of Web services [19]. Cristian Mateos et al. analyzed
Copyright © 2017 for this paper by its authors. 66
5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2017)
the available approaches to remove undesirable anti-patterns LSSVM based approach. The Web Service dataset provides the
using code-first [12]. Ping Wang proposed another decision quality of service parameters values such as response time,
model under obscure data to choose a Web Service [21]. availability, throughput, compliance, latency for 2507 Web
Services. The QoS parameters values by the dataset provider
III. R ESEARCH F RAMEWORK
are computed using Web service benchmark tools [1]. In this
A. Dependent Variables- QoS Parameters study, we use 200 Web Services for the analysis. The reason
Al-Masri et al. define 9 quality of service parameters of for selection of 200 web-services is stated in our earlier work
Web Services. They compute the QoS parameters using Web [7] as the study presented in this paper is an extension of the
service benchmark tools. The QoS parameters are: Availability previous work.
(AV), Best Practices (BP), Compliance (CP), Documentation
D. Feature Extraction using Principal Component Analysis
(DOC), Latency (LT), Response Time (RT), Reliability (REL),
(PCA)
Successability (SA), Throughput (TP), Maintainability, Modu-
larity, Reusability, Testability, Interoperability and Conformity. We perform feature extraction using Principal Component
These QoS parameters are the dependent variables for the Analysis (PCA). The main motivation of using PCA is for
machine learning model. transforming high dimension data space into lower dimension
data space. The lower dimension data consists of the most
B. Predictor Variables: Source Code Metrics significant features [20]. We label the new metrics (or features)
Three different types of metrics suite are considered as after applying PCA as principal component domain metrics.
independent or predictor variables Figure 1 displays the steps followed by us to extract the feature
1) Object-Oriented Source Code Metrics: We compute set using PCA.
nineteen different Object-Oriented source code metrics from
the bytecode of the compiled Java files of the Web Services Eigen value and Eigen
in our experimental dataset using CKJM extended tool1 [5]. Feature vector computation
using MATLAB
normalization -
CKJM extended is an extended version of tool for calculating zero mean value command
Data Set (eign = eig(data))
Chidamber and Kemerer Java source code metrics and many
other metrics such as average method complexity, McCabe’s
Cyclomatic Complexity, lack of cohesion among the classes.
Java class files from the WSDL file are generated using Principal components
Reduced set of
WSDL2Java Axis2 code generator2 , which is available as an features (metrics)
selection based on
eigenvalue being greater
Eclipse plug-in. We then compiled the Java files to generate are evaluated
than 1.0
the bytecode for computing the size and structure of software
metrics using the CKJM extended tool. Fig. 1: Sequence of Steps for Applying PCA
2) Henry M. Sneed WSDL Metric Suite: Sneed et al.
develop a tool for measuring Web Service interfaces [17][6]. We apply PCA with varimax rotation technique on all the
The suite primarily consists of six different source code software metrics. Table I shows the result and outcome of PCA
metrics to measure complexity of service interfaces: Data with varimax rotation method. Table I reveals the relationship
Flow Complexity, Interface Relation Complexity, Interface between domain metrics and original software metrics. For
Data Complexity, Interface Structure Complexity, Interface each principal component (PC), % variance, % cumulative and
Format Complexity and Language Complexity. These metrics interpreted metrics set are presented in Table I.
are statically computed from a service interface in WSDL. The TABLE I: Principle Component Analysis (PCA) Results
metrics are based on analyzing the WSDL schema elements
PC Eigenvalue variance % % Cumulative Interpreted Metrics
[17][6]. PC1 6.40 17.30 17.30 Ce, Ca, RFC, CBO, LCO, LCOM3,
CAM, DAM
3) Baski and Misra Metrics: Baski and Misra proposed a PC2 5.8 15.76 33.06 DP, FP, OP, MRS, OPS, IDFC, IRC
tool to compute six different complexity metrics of WSDL file PC3 3.67 9.94 43.00 CE, MiRV, MDC, MeRV, DW, MR
PC4 3.39 9.16 52.17 ILC, DMR, ISC, IDC
[4]. These metrics are based on the analysis of the structure PC5 3.34 9.03 61.2 MOA, CBM, IC
PC6 2.50 6.77 67.98 MFA, NOC, DIT, IFC
of the exchanged messages described in WSDL file which PC7 2.23 6.02 74.00 NPM, WMC
becomes the basis for computing the data complexity. These PC8 2.14 5.79 79.79 AMC, MRV
PC9 1.36 3.7 83.5 LCOM
metrics are based on analyzing the WSDL and XSD schema
elements [4].
E. Feature Selection using Rough Set Analysis (RSA)
C. Experimental Dataset
Before the application of RSA, the input data need to
In our study, the Web Service dataset collected by Al-Masri be categorized. In our study, K-means clustering approach
et al. 3 is used to measure the performance of the proposed is applied for the purpose of data categorization. In the
1 http://gromit.iiar.pwr.wroc.pl/p inf/ckjm/ approach, the data belonging to a particular cluster are grouped
2 https://sourceforge.net/projects/wsdl2javawizard/ under a single category or class. After the application of K-
3 http://www.uoguelph.ca/∼qmahmoud/qws/ means clustering approach, we obtain 3 clusters and the data
Copyright © 2017 for this paper by its authors. 67
5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2017)
were categorized into three groups: High, Medium, and Low
Availability
correlation. Figure 1 shows the steps followed to identify the
best set (in-terms of relevance) of features using RSA. AM Response Time
Feature Successability
Calculation of Lower
Categorization into and upper
groups using approximation of all
k-means clustering BMS Throughput
Data Set method
possible sets
Compliance
Identification of Computation of Reliability
HMS
best set of Accuracy for all
source-code metrics possible sets
Latency
Fig. 2: Rough Set Analysis (RSA) based Feature Selection
OOM Best Practices
Table II displays the selected set of source-code metrics for
every QoS parameter. The resultant set is derived from the Maintainability
original set of 37 source-code metrics using RSA approach. PCA
Documentation
TABLE II: Source Code Metrics Identified using Rough Set RSA Reusability
Analysis (RSA)
QoS Selected Metrics Modularity
Availability MiRV, Ca, CC, CAM, IC, MFA, LC, SC, LCOM3, WMC, FC, MeRV
Response Time Ca, DMR, LC, SC, WMC, MFA, CC, IC, CAM, LCOM3
Successability CAM, LCOM3, DAM, FC, LC, DFC, MRV, ME, SC, WMC, LCO, MOA
Independent Variables
Throughput MiRV, Ce, CC, CAM, ME, MFA, LC, SC, CBM, MRV, FC, MeRV, MOA Interoperability
Compliance MiRV, NPM, CC, WMC, CAM, MOA, SC, LC, FC, DFC, ME, Ca, MRV, DAM
Reliability LCOM3, MFA, FC, LC, DFC, CAM, SC, WMC, LCO, MOA
Latency IC, FC, LC, DMR, MRV, MOA, ME, CAM, DC, DFC, NOC, LCO, NPM
Best Practices MiRV, Ca, CC, CAM, ME, MFA, LC, SC, MRV, FC, MOA, WMC, DFC, NPM Testability
Maintainability CBM, DP, LCOM3, MFA, Ce, CAM, MOA
Documentation CC, LC, ME, IC, SC, CAM, Ca, DFC, MRV, WMC, MeRV, FC, NPM
Reusability LCOM3, FC, MDC, LCOM, DMR, SC, LC, DFC
Modularity AMC, LC, DMR, SC, Ca, IC, DFC, ME, DP, MiRV, MOA, MRV, WMC Conformity
Interoperability MiRV, CC, SC, MeRV, LC, WMC, MFA, DIT, CBO
Testability FC, CC, RFC, ME, NOC, MiRV, DIT, SC, LC
Conformity CAM, Ca, ME, DFC, FC, WMC, MRV, LC Dependent Variables
Fig. 3: Dependent and Independent Variables
F. Effectiveness of Metrics
Once we have the QoS data, the relationship or degree of
association between source code metrics and QoS parame- generate models for predicting QoS parameters. The different
ters can be determined. The set of source code metrics are set of source code metrics are used a input of the models. We
considered as independent variables and QoS parameters are also examine LSSVM different kernel functions to investigate
considered as a dependent variables. In our experiments, six if we can achieve better result and compare the performance
different set of source metrics (all metrics (AM), Baski and of various kernel functions.
Misra metrics suite, Henry M. Sneed WSDL metric suite, The block diagram displayed in Figure 4, illustrates the
object-oriented metrics suite, selected set of metrics princi- sequence of steps used to determine the predicted quality of
pal component analysis (PCA), and selected set of metrics service (QoS) parameters using the LSSVM method with three
using rough set analysis (RSA) are considered as input to different types of kernel functions. In our work, the following
develop fifteen QoS parameters (Response Time Availabil- steps are performed to generate quality of service prediction
ity, Throughput, Successability, Reliability, Compliance, Best models:
Practices, Latency, Documentation, Maintainability, Modular- 1) Source code metrics computation for all the Web Ser-
ity, Reusability, Testability, Interoperability, and Conformity) vices in the data-set as described in Section III-B.
prediction models. Figure 3 shows the independent and de- 2) Selection of suitable set of source code metrics using
pendent variables used for QoS parameter prediction model. PCA and RSA feature extraction and selection tech-
From Figure 3, we infer that a total of eight different sets of niques.
independent variables are possible for each QoS parameter. 3) Predictive model generation by considering source code
metrics as input to estimate fifteen different quality of
IV. P ROPOSED M ACHINE L EARNING BASED A PPROACH service parameters.
Least Square Support Vector Machines (LSSVM) are super- 4) Identification of performance measures to evaluate the
vised learning methods having wide range of applications doe predictive ability and effectiveness of quality of service
classification, regression and outliers detection problems [18]. prediction models.
In our experiments, we use LSSVM as regression technique to 5) Application of validation methods to determine the true
Copyright © 2017 for this paper by its authors. 68
5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2017)
Baski and Misra Performance
Metrics (BMS) Linear Kerenl
Evaluation
Baski and
Baski and
Misra Metric
Misra Metrics
suite
Henry M. Sneed
WSDL File Metrics (HMS)
Henry M.
Sneed Tool Polynomial
Sneed Metrics Statistical Test
Object Kerenl
Oriented
QoS metrics
WSDL to Java (OOM)
Object- All metrics (AM)
CKJM
Source Code Oriented Model
Extended RBF Kerenl
Metrics PCA comparsion
RSA
Set of source code metrics Least Square Support Vector Validation of developed models
Machines (LSSVM method)
Fig. 4: Proposed Steps Used for the QoS Prediction (A Multi-Step Process)
predictive applicability of the predictive models. D. Performance Parameter
6) Application of rigorous statistical significance tests to In order to evaluate the QoS prediction model, various
compare the performance of one prediction technique performance parameters are defined in the machine learning
over other approaches and also determine the superiority literature to measure the effectiveness of the QoS prediction
of one set of source code metrics over the other sets. models. In this work, we consider three different performance
parameters: Mean Magnitude Relative Error (MMRE), Mean
A. Computation of Source Code Metrics
Absolute Error (MAE), and Root Mean Square Error (RMSE)
During our experimental analysis, we consider 200 Web Ser- to evaluate the QoS prediction model [13]. A lower value for
vices as the experimental dataset (same as used by Kumar et these performance parameters denotes an effective prediction
al. [7]). In this work, WSDL interface complexity metrics and model.
WSDL complexity metrics are computed using Henry Sneed
metrics tool and Baski and Misra metrics tool respectively. E. Validation Method
Then we use the wsimport 4 tool to parse WSDL document file The objective of the study presented in this paper is to
of a Web Service and generate its corresponding Java class. build and apply statistical models to predict different QoS
This involves extracting the Java source code implementing parameters for future releases and unseen similar natured
the service. As shown in Figure 4, we compiled the Java files projects. Hence, it is necessary to validate the developed QoS
to generate the bytecode for computing the size and structure model on a different data-sets than the dataset on which the
software metrics using the CKJM extended tool. training is done. In our experiments, we consider the standard
k-fold cross validation approach (we take k = 10) to validate
B. Feature Selection Method the proposed QoS model. In our analysis, we also perform
In our experiments, we examine 37 different software outlier detection analysis to eliminate the extreme values
metrics (Chidamber and Kemerer, Harry M. Sneed, Baski which may add to the noise & bias in the model performance
& Misra)) to predict 15 different QoS attributes. It is very and accuracy results. The outlier analysis is doen based on the
essential to remove irrelevant and unimportant source code following equations:
metrics out of these source code metrics so that only relevant
source code metrics are included in the construction of QoS if |yji − yˆj | > 3 ∗ σ for Effective outliers
ei =
prediction models. In order to achieve the stated objective, we if |yji − yˆj | <= 3 ∗ σ for Non Effective outliers
consider two different features selection techniques: principal (1)
component analysis and rough set analysis for feature selec-
F. Statistical Significance Tests and Procedures
tion.
In order to bring rigour and mitigate threats to validity in
C. Prediction Techniques our analysis of the results, we apply pairwise t-test approach.
We conduct t-test to determine which prediction method and
In this work, we have use LSSVM with three different
feature selection techniques performs relatively better or does
kernel functions to develop QoS prediction model.
the models perform equally well. We analyze all the results
4 http://docs.oracle.com/javase/6/docs/technotes/ based on the 0.05 significance level, i.e. two models are
tools/share/wsimport.html significantly different (null hypothesis rejected) if the p-value
Copyright © 2017 for this paper by its authors. 69
5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2017)
is less than 0.05 (the cut-off valie) else there is no significant Table III displays the MMRE, MAE, and RMSE values
different between them (null hypothesis accepted). obtained after applying LSSVM method with linear kernel,
polynomial kernel and RBF kernel functions. Table III pro-
0.4
vides complete and detailed results for all the QoS parameters,
feature extraction and selecion techniques, metrics suites and
0.3
performance evaluation methods. From Table III, we make the
0.2
following observations:
Pearson residual
0.1
0 1) In case of linear kernel function, we observe that the
-0.1 model built by considering selected set of metrics using
-0.2 RSA as input has low values of MMRE, MAE and
-0.3 RMSE in comparison with other sets of metrics. This
-0.4
clearly implies that the performance of the model de-
-0.5
veloped using RSA is much better than the performance
BMS HMS OOM AM PCA RSA of other models i.e., low vales of MMRE, MAE, and
Fig. 5: Box-Plot Visual Analysis (Linear Kernel) RMSE for QoS prediction as compared to a model
developed using other sets of metrics.
2) In case of polynomial kernel function, we observe that
the model built by considering all metrics has low value
of MMRE, MAE and RMSE in comparison to other
0.3
sets of metrics. We thus infer that the performance of
0.2
the prediction model developed using all metrics (AM) is
Pearson residual
0.1
much better than other models i.e., low vales of MMRE,
0
MAE, and RMSE for QoS prediction as compared to a
-0.1 model developed using other sets of metrics.
-0.2 3) In case of RBF kernel function, we notice that the model
-0.3 developed by considering Baski and Misra Metric has
-0.4 low value of MMRE, MAE, and RMSE in comparison
with other sets of metrics. This implies that the per-
BMS HMS OOM AM PCA RSA
formance of the model developed using BMS is much
Fig. 6: Box-Plot Visual Analysis (Polynomial Kernel) better than other models i.e., low vales of MMRE, MAE,
and RMSE for QoS prediction as compared to a model
developed using other sets of metrics.
0.8
In order to have a visual comparion (visual analytics) of
the results, we draw the Pearson residual boxplots of models
0.6
developed using LSSVM method with linear kernel function.
Pearson residual
0.4
The plots are displayed in Figure 5. The middle line of each
0.2 box in Figure 5 shows the median value of Pearson residual.
0 From Figure 5, we observe that all models built have a median
-0.2 residual value close to zero and the model developed by
-0.4 considering selected set of metrics using RSA has smallest
-0.6
whiskers, narrowest box as well as few numbers of outliers.
This shows that model developed by considering selected set
BMS HMS OOM AM PCA RSA of metrics using RSA as input results in better performance
as compared to other metrics. Figure 6 shows the Pearson
Fig. 7: Box-Plot Visual Analysis (RBF Kernel)
residual boxplots of models developed using LSSVM method
with polynomial kernel function. From Figure 6, we infer that
V. E XPERIMENTAL A NALYSIS all models have a median residual value close to zero and
In this paper, LSSVM method with three different types of the model developed based on selected set of metrics using
kernel functions have been considered to develop a model to AM has smallest whiskers, narrowest box as well as few
predict fifteen different QoS parameters by considering six numbers of outliers. This result shows that model developed
different set of source code metrics as input. The detailed by considering AM as input results in better performance as
descriptions of these source code metrics were already pre- compared to others.
sented in section III. The performance of the developed QoS Figure 7 shows the Pearson residual boxplots of models
prediction models are compared using three different types of developed using LSSVM method with RBF kernel function.
performance parameters i.e., MMRE, MAE and RMSE. From Figure 7, it is observed that all developed models have
Copyright © 2017 for this paper by its authors. 70
5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2017)
TABLE III: Performance Matrix for LSSVM method with Linear, Polynomial and RBF Kernel and All Different Metrics Suites
ity
ility
ime
ion
ces
ity
il
ntat
t
erab
nce
se T
nab
ity
ility
ty
hpu
abil
lity
acti
ity
ity
i
form
ular
plia
ume
ency
sabi
ntai
abil
abil
t Pr
ilab
cess
oug
rop
pon
Com
Mod
Con
Test
Reli
Mai
Reu
Inte
Thr
Doc
Ava
Res
Suc
Lat
Bes
Linear Kernel
MMRE
AM 0.32 0.32 0.35 0.74 0.60 0.68 0.56 0.46 0.32 0.90 0.29 0.14 0.16 0.13 0.06
OOM 0.31 0.33 0.34 0.73 0.59 0.69 0.51 0.36 0.39 0.89 0.46 0.44 0.42 0.49 0.25
WIM 0.32 0.34 0.35 0.74 0.61 0.67 0.56 0.38 0.49 0.92 0.42 0.22 0.20 0.18 0.06
WSM 0.31 0.33 0.35 0.75 0.60 0.68 0.56 0.39 0.72 0.91 0.66 0.41 0.35 0.43 0.21
RSA 0.31 0.34 0.34 0.65 0.60 0.69 0.56 0.34 0.84 0.94 0.62 0.32 0.38 0.30 0.10
PCA 0.31 0.34 0.35 0.74 0.61 0.66 0.55 0.37 0.54 0.92 0.61 0.33 0.31 0.33 0.13
MAE
AM 0.13 0.09 0.16 0.19 0.25 0.24 0.11 0.19 0.08 0.21 0.09 0.04 0.08 0.04 0.03
OOM 0.13 0.09 0.16 0.18 0.25 0.24 0.1 0.17 0.10 0.21 0.13 0.12 0.18 0.13 0.14
WIM 0.14 0.10 0.17 0.18 0.26 0.24 0.11 0.18 0.11 0.22 0.11 0.06 0.10 0.04 0.03
WSM 0.14 0.10 0.17 0.18 0.26 0.24 0.11 0.18 0.17 0.22 0.16 0.11 0.15 0.11 0.13
RSA 0.13 0.10 0.16 0.16 0.25 0.24 0.11 0.17 0.20 0.23 0.15 0.10 0.18 0.10 0.06
PCA 0.14 0.10 0.17 0.18 0.26 0.23 0.11 0.18 0.13 0.22 0.15 0.10 0.14 0.09 0.08
RMSE
AM 0.20 0.13 0.22 0.22 0.29 0.27 0.14 0.23 0.11 0.26 0.11 0.05 0.1 0.05 0.04
OOM 0.19 0.14 0.21 0.22 0.29 0.27 0.13 0.21 0.13 0.27 0.17 0.17 0.22 0.18 0.17
WIM 0.20 0.15 0.22 0.21 0.30 0.27 0.15 0.22 0.16 0.28 0.15 0.08 0.13 0.07 0.05
WSM 0.20 0.15 0.22 0.21 0.29 0.27 0.15 0.22 0.22 0.28 0.21 0.15 0.20 0.16 0.16
RSA 0.19 0.15 0.22 0.2 0.29 0.27 0.14 0.2 0.25 0.29 0.20 0.15 0.23 0.14 0.10
PCA 0.20 0.15 0.22 0.21 0.30 0.26 0.14 0.21 0.17 0.28 0.19 0.14 0.18 0.13 0.11
Polynomial Kernel
MMRE
AM 0.32 0.33 0.35 0.74 0.61 0.68 0.56 0.48 0.73 0.90 0.70 0 0.37 0 0.25
OOM 0.31 0.33 0.34 0.73 0.59 0.70 0.55 0.38 1.02 0.89 0.61 0.59 0.51 0.53 0.37
WIM 0.32 0.34 0.35 0.74 0.61 0.67 0.56 0.38 0.5 0.92 0.62 0.11 0.19 0.19 0.54
WSM 0.31 0.33 0.34 0.73 0.60 0.67 0.56 0.39 0.70 0.91 0.71 0.54 0.32 0.48 0.15
RSA 0.3 0.34 0.34 0.74 0.61 0.69 0.56 0.36 0.85 0.94 0.62 0.47 0.47 0.32 0.60
PCA 0.31 0.34 0.35 0.74 0.61 0.68 0.55 0.38 1.03 0.92 0.65 0.45 0.36 0.38 0.14
MAE
AM 0.13 0.09 0.16 0.19 0.26 0.24 0.11 0.20 0.18 0.21 0.17 0 0.16 0 0.11
OOM 0.13 0.09 0.16 0.18 0.25 0.24 0.11 0.18 0.24 0.22 0.15 0.16 0.23 0.14 0.20
WIM 0.14 0.10 0.17 0.18 0.26 0.24 0.11 0.18 0.12 0.22 0.15 0.03 0.09 0.05 0.24
WSM 0.14 0.10 0.17 0.18 0.26 0.24 0.11 0.18 0.16 0.22 0.16 0.14 0.13 0.12 0.09
RSA 0.13 0.10 0.16 0.18 0.26 0.24 0.11 0.18 0.21 0.23 0.15 0.14 0.22 0.10 0.27
PCA 0.14 0.10 0.17 0.18 0.26 0.24 0.11 0.18 0.25 0.22 0.15 0.13 0.16 0.11 0.08
RMSE
AM 0.20 0.14 0.22 0.22 0.29 0.28 0.14 0.23 0.22 0.26 0.22 0 0.20 0 0.14
OOM 0.19 0.14 0.21 0.22 0.29 0.28 0.14 0.22 0.30 0.27 0.20 0.22 0.27 0.20 0.22
WIM 0.20 0.15 0.22 0.21 0.30 0.27 0.15 0.22 0.16 0.28 0.19 0.05 0.12 0.08 0.25
WSM 0.20 0.14 0.22 0.21 0.29 0.27 0.15 0.22 0.22 0.28 0.22 0.19 0.18 0.18 0.12
RSA 0.19 0.15 0.22 0.21 0.30 0.28 0.14 0.21 0.26 0.29 0.20 0.19 0.26 0.14 0.29
PCA 0.20 0.15 0.22 0.21 0.30 0.27 0.14 0.21 0.30 0.28 0.20 0.17 0.19 0.15 0.1
RBF Kernel
AM 0.32 0.32 0.35 0.71 0 0.67 0.38 0.45 0.09 0.38 0.08 0.03 0.08 0.05 0.02
OOM 0.3 0.32 0.34 0.57 0.13 0.68 0.28 0.36 0.22 0.01 0.40 0.42 0.20 0.44 0.13
WIM 0.32 0.34 0.35 0.27 0.42 0.39 0.45 0.29 0.08 0.92 0.16 0.03 0.07 0.02 0.02
WSM 0.3 0.33 0.33 0.50 0.60 0.67 0.56 0.37 0.39 0.89 0.30 0.35 0.16 0.24 0.05
RSA 0.3 0.34 0.34 0.60 0 0.68 0.28 0.35 0.58 0.34 0.10 0 0.16 0.13 0.09
PCA 0.31 0.33 0.35 0.73 0 0.67 0.49 0.36 0.17 0.29 0.27 0.18 0.21 0.22 0.05
MAE
AM 0.13 0.09 0.16 0.18 0 0.23 0.07 0.18 0.03 0.09 0.03 0.01 0.04 0.02 0.01
OOM 0.13 0.09 0.16 0.14 0.06 0.24 0.05 0.17 0.06 0 0.11 0.12 0.09 0.11 0.07
WIM 0.14 0.10 0.17 0.07 0.18 0.14 0.09 0.14 0.03 0.22 0.04 0.01 0.04 0.01 0.01
WSM 0.13 0.10 0.16 0.12 0.26 0.23 0.11 0.18 0.10 0.22 0.07 0.09 0.06 0.06 0.02
RSA 0.13 0.10 0.16 0.15 0 0.24 0.06 0.17 0.14 0.08 0.03 0 0.07 0.04 0.05
PCA 0.14 0.10 0.17 0.18 0 0.23 0.09 0.17 0.05 0.07 0.07 0.05 0.09 0.06 0.03
RMSE
AM 0.20 0.14 0.22 0.21 0 0.27 0.10 0.22 0.04 0.11 0.04 0.01 0.05 0.02 0.02
OOM 0.19 0.14 0.21 0.17 0.07 0.27 0.07 0.21 0.08 0.01 0.15 0.17 0.12 0.16 0.10
WIM 0.20 0.15 0.22 0.08 0.20 0.16 0.12 0.17 0.04 0.28 0.06 0.02 0.05 0.01 0.02
WSM 0.19 0.15 0.21 0.15 0.29 0.27 0.15 0.21 0.14 0.27 0.10 0.14 0.10 0.09 0.04
RSA 0.19 0.15 0.22 0.18 0 0.27 0.07 0.20 0.19 0.11 0.03 0 0.09 0.07 0.08
PCA 0.20 0.15 0.22 0.21 0 0.27 0.13 0.20 0.06 0.09 0.10 0.08 0.12 0.09 0.04
Copyright © 2017 for this paper by its authors. 71
5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2017)
TABLE IV: Result of t-test: Among Different Metrics Set
P-Value
MMRE MAE RMSE
AM OOM HMS BMS RSA PCA AM OOM HMS BMS RSA PCA AM OOM HMS BMS RSA PCA
AM 1.000 0.063 0.031 0.063 0.031 0.031 1.000 0.031 0.031 0.031 0.031 0.031 1.000 0.031 0.031 0.031 0.031 0.031
OOM 0.063 1.000 0.031 0.031 0.031 0.031 0.031 1.000 0.031 0.031 0.031 0.031 0.031 1.000 0.031 0.031 0.031 0.031
HMS 0.031 0.031 1.000 0.031 0.094 0.031 0.031 0.031 1.000 0.063 0.031 0.031 0.031 0.031 1.000 0.313 0.031 0.031
BMS 0.063 0.031 0.031 1.000 0.031 0.031 0.031 0.031 0.063 1.000 0.031 0.031 0.031 0.031 0.313 1.000 0.031 0.031
RSA 0.031 0.031 0.094 0.031 1.000 0.031 0.031 0.031 0.031 0.031 1.000 0.031 0.031 0.031 0.031 0.031 1.000 0.031
PCA 0.031 0.031 0.031 0.031 0.031 1.000 0.031 0.031 0.031 0.031 0.031 1.000 0.031 0.031 0.031 0.031 0.031 1.000
Mean Difference
MMRE MAE RMSE
AM OOM HMS BMS RSA PCA AM OOM HMS BMS RSA PCA AM OOM HMS BMS RSA PCA
AM 0.000 0.020 -0.392 -0.013 -0.345 -0.268 0.000 0.038 -0.082 -0.068 -0.142 -0.158 0.000 -0.052 -0.067 -0.073 -0.123 -0.148
OOM -0.020 0.000 -0.412 -0.033 -0.365 -0.288 -0.038 0.000 -0.043 -0.030 -0.103 -0.120 0.052 0.000 -0.015 -0.022 -0.072 -0.097
HMS 0.392 0.412 0.000 0.378 0.047 0.123 0.082 0.043 0.000 0.013 -0.060 -0.077 0.067 0.015 0.000 -0.007 -0.057 -0.082
BMS 0.013 0.033 -0.378 0.000 -0.332 -0.255 0.068 0.030 -0.013 0.000 -0.073 -0.090 0.073 0.022 0.007 0.000 -0.050 -0.075
RSA 0.345 0.365 -0.047 0.332 0.000 0.077 0.142 0.103 0.060 0.073 0.000 -0.017 0.123 0.072 0.057 0.050 0.000 -0.025
PCA 0.268 0.288 -0.123 0.255 -0.077 0.000 0.158 0.120 0.077 0.090 0.017 0.000 0.148 0.097 0.082 0.075 0.025 0.000
TABLE V: t-test: Among different Kernel Hence, null hypothesis is rejected only if the p-value is less
P-Value than 0.05
3 = 0.0167. The null hypothesis while applying the
MMRE MAE RMSE
Lin Poly RBF Lin Poly RBF Lin Poly RBF
Wilcoxon test is that there is no significant difference between
Lin 1.000 0.000 0.000 1.000 0.000 0.000 1.000 0.000 0.000 the two classification techniques. From the result Table V, we
Poly 0.000 1.000 0.000 0.000 1.000 0.000 0.000 1.000 0.000
RBF 0.000 1.000 0.000 0.000 1.000 0.000 0.000 1.000 0.000 observe that there is a significant difference between the
Mean Difference kernel functions. This interpretation is due to the fact that
MMRE MAE RMSE
Lin Poly RBF Lin Poly RBF Lin Poly RBF the p-value is lower than 0.0167 (rejecting the null hypothesis
Lin 0.000 -0.051 0.155 0.000 -0.015 0.047 0.000 -0.016 0.057 and accpeting the alternate hypothesis). However by closely
Poly 0.051 0.000 0.206 0.015 0.000 0.063 0.016 0.000 0.074
RBF -0.155 -0.206 0.000 -0.047 -0.063 0.000 -0.057 -0.074 0.000 examining the value of mean difference, RBF kernel function
yields better result as compared to other kernel functions.
a median residual value being close to zero and the model
B. Source Code Metrics Sets
developed by considering selected set of metrics using BMS
has smallest whiskers, narrowest box as well as few numbers In this work, six different set of source code metrics are
of outliers. This shows that model developed by considering used as input to develop QoS prediction models. Hence for
BMS as input obtained better performance as compared to each set of source code metrics, a total of three sets (one for
others. each performance parameter) are used. Each set has 45 data
points (15 QoS parameters * 3 kernel functions). Results of
VI. COMPARISON OF VARIOUS K ERNEL F UNCTIONS AND Wilcoxon test with Bonferroni correction for all performance
M ETRIC S UITES parameter is shown in Table V. The Bonferroni correction sets
We apply Wilcoxon signed rank test to compare the per- the significance cutoff at αn , where n is number of different
formance of the models using LSSVM method with three pairs (here 6 sets of metrics; n=6technique C2 = 6 ∗ 5/2 = 15)
different types of kernel functions and different sets of source and all results are analyzed at a 0.05 significance level. Hence,
code metrics. We use Wilcoxon test with Bonferroni correction null hypothesis is rejected only if the p-value is less than
0.05
for comparative analysis. 15 = 0.0033. From Table IV, we infer that there is no
significant difference between sets of metrics. We arrive
A. Kernel Functions at this conclusion due to the fact that the p-value is greater
Three different types of kernel functions have been applied than 0.0033 (accepting the null hypothesis). However, by
to develop QoS prediction models. Hence for each of the closely examining the value of mean difference, we infer that
kernel functions, a total number of three sets (one for each per- the object-oriented Metrics are yields better performance
formance parameter) are used. Each set has 90 data points (15 results in comparison to other sets of metrics.
QoS parameters * 6 sets of metrics). The results of Wilcoxon
test with Bonferroni correction for all performance parameters VII. T HREATS TO VALIDITY
are shown in Table V. The upper portion of the Table V shows
the p-value between kernel functions and the lower portion One threat to validity is that the impact on the dependent
shows the mean difference value of performance parameters variable may not be completely attributed to the changes in
between different kernels. The Bonferroni correction sets the the independent variable due to overfitting of the predictive
significance cut-off at α n , where n is number of different model. Another threat to validity is that we have conducted
pairs (3 kernel functions; n=3technique C2 = 3 ∗ 2/2 = 3) experiments on a limited dataset and the answer result can be
and all results are analyzed at a 0.05 significance level. biased to the specific dataset.
Copyright © 2017 for this paper by its authors. 72
5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2017)
VIII. C ONCLUSION to predict maintainability of service oriented software.
Our main conclusions and takeaways are the following: In High Assurance Systems Engineering (HASE), 2017
• Based on our correlation analysis of the metrics, we
IEEE 18th International Symposium on, pages 88–95.
conclude that there exists a high correlation between IEEE, 2017.
Object-Oriented metrics and WSDL metrics. [9] L. Kumar, S. K. Rath, and A. Sureka. Using source code
• From the result of Wilcoxon test with Bonferroni correc-
metrics to predict change-prone web services: A case-
tion, we conclude that there is a statistically significant study on ebay services. In Machine Learning Techniques
difference between the performance of the predictive for Software Quality Evaluation (MaLTeSQuE), IEEE
models built using three different LSSVM kernel func- Workshop on, pages 1–7. IEEE, 2017.
tions. [10] W. Li and S. Henry. Maintenance metrics for the Object-
• From the rest of Wilcoxon test with Bonferroni correc-
Oriented paradigm. In International Software Metrics
tion, we conclude that there is no statistically significant Symposium, pages 52–60, 1993.
difference between different sets of source code metrics. [11] R. Malhotra and Y. Singh. On the applicability of
• From Table III, we conclude that no one set of source-
machine learning techniques for object oriented software
code metrics dominate the other sets for any QoS param- fault prediction. Software Engineering: An International
eter and vice-versa. Journal.
• By assessing the value of mean difference shown in Table
[12] C. Mateos, M. Crasso, A. Zunino, and J. L. O. Cos-
V, we conclude that that the RBF kernel for LSSVM cia. Detecting wsdl bad practices in code–first web
method yields better performance results compared to services. International Journal of Web and Grid Services,
other kernel functions. 7(4):357–387, 2011.
• By assessing the value of mean difference shown in
[13] T. Menzies, Z. Chen, J. Hihn, and K. Lum. Selecting
Table IV, we conclude that the object-oriented metrics best practices for effort estimation. IEEE Transactions
yields better result compared to other sets of source code on Software Engineering, 32(11):883–895, 2006.
metrics. [14] M. Perepletchikov, C. Ryan, and K. Frampton. Cohesion
• Our analysis shows evidences that it is possible to esti-
metrics for predicting maintainability of service-oriented
mate the QoS parameters of Web Services using source software. In QSIC, pages 328–335. IEEE, 2007.
code metrics and LSSVM based method. [15] M. Perepletchikov, C. Ryan, K. Frampton, and Z. Tari.
Coupling metrics for predicting maintainability in
R EFERENCES service-oriented designs. In ASWEC, pages 329–340.
[1] E. Al-Masri and Q. H. Mahmoud. Investigating web IEEE, 2007.
services on the world wide web. In Proceedings of the [16] B. Shim, S. Choue, S. Kim, and S. Park. A design quality
17th international conference on World Wide Web, pages model for service-oriented architecture. In 2008 15th
795–804. ACM, 2008. Asia-Pacific Software Engineering Conference, pages
[2] V. R. Basili, L. C. Briand, and W. L. Melo. How 403–410. IEEE, 2008.
reuse influences productivity in object-oriented systems. [17] H. M. Sneed. Measuring web service interfaces. In Web
Communications of the ACM. Systems Evolution (WSE), pages 111–115. IEEE, 2010.
[3] V. R. Basili, L. C. Briand, and W. L. Melo. A [18] J. A. Suykens, J. De Brabanter, L. Lukas, and J. Vande-
validation of Object-Oriented design metrics as quality walle. Weighted least squares support vector machines:
indicators. IEEE Transactions on Software Engineering, robustness and sparse approximation. Neurocomputing.
22(10):751–761, October 1996. [19] V. X. Tran, H. Tsuji, and R. Masuda. A new qos ontology
[4] D. Baski and S. Misra. Metrics suite for maintainability and its qos-based ranking algorithm for web services.
of extensible markup language web services. IET Soft- Simulation Modelling Practice and Theory, 17(8):1378–
ware, 5(3):320–341, 2011. 1398, 2009.
[5] S. R. Chidamber and C. F. Kemerer. A metrics suite for [20] D. Wang and J. Romagnoli. Robust multi-scale prin-
Object-Oriented design. IEEE Transactions on Software cipal components analysis with applications to process
Engineering, 20(6):476–493, June 1994. monitoring. Journal of Process Control, 15(8):869–882,
[6] J. L. O. Coscia, M. Crasso, C. Mateos, and A. Zunino. 2005.
Estimating web service interface quality through conven- [21] P. Wang. Qos-aware web services selection with in-
tional object-oriented metrics. CLEI Electron. J, 16(1), tuitionistic fuzzy set under consumer’s vague percep-
2013. tion. Expert Systems with Applications, 36(3):4460–
[7] L. Kumar, S. K. Rath, and A. Sureka. Predicting qual- 4466, 2009.
ity of service (qos) parameters using extreme learning [22] Y. Zhou and H. Leung. Empirical analysis of object-
machines with various kernel methods. In Quantitative oriented design metrics for predicting high and low
Approaches to Software Quality, page 11, 2016. severity faults. 32(10):771–789, 2006.
[8] L. Kumar, S. K. Rath, and A. Sureka. Using source
code metrics and multivariate adaptive regression splines
Copyright © 2017 for this paper by its authors. 73