I. RESEARCH MOTIVATION AND AIM
Web services are distributed web application components
which can be implemented in different languages, deployed
on different client and server platforms, are represented by
interfaces and communicate using open protocols [
1
][
2
]. Web
service implementers and providers need to comply with
common web service standards so that they can be language and
platform independent and can be discovered and used by other
applications [
1
][
2
]. Applications and business solutions using
web services (which integrate and combine several services)
expect high Quality of Service (QoS) such as performance,
scalability and reliability as their application is dependent
on the service. Measuring quality of service attributes and
characteristics of web services and understanding their
relationship with source code metrics can help developers control
and estimate maintainability by analyzing the source code
[
3
][
4
][
5
][
6
]. The work presented in this paper is motivated by
the need to investigate the correlation between QoS attributes
such as response time, availability, throughput, reliability,
modularity, testability and interoperability and source code
metrics such as classic object oriented metrics (Chidamber and
Kemerer) as well as other well-known metrics such as Baski &
Misra and Harry M. Sneed metrics. Specifically, our research
aim is to study the correlation between 15 web service quality
attributes and 37 source code metrics and then build machine
learning based predictive models for estimating the quality of
a given service based on the computed source code metrics.
Our aim is to conduct experiments on a real-world dataset and
also examine the extent to which feature selection techniques
such as Principal Component Analysis (PCA) and Rough Set
Analysis (RSA) can be used for dimensionality reduction and
filter irrelevant features.
II. RELATED WORK, RESEARCH CONTRIBUTIONS AND
RESEARCH FRAMEWORK
Related Work: Coscia et al. investigate the potential of
obtaining more maintainable services by exploiting Object-Oriented
metrics (OO) values from the source code implementing
services [
3
]. Their approach proposed the use of OO metrics as
early indicators to guide software developers towards obtaining
more maintainable services [
3
]. Coscia and Crasso et al.
present a statistical correlation analysis demonstrating that
classic software engineering metrics (such as WMC, CBO,
RFC, CAM, TPC, APC and LCOM) can be used to predict the
most relevant quality attributes of WSDL documents [
4
].
Mateos et al. found that there is a high correlation between
wellknown object-oriented metrics taken in the code implementing
services and the occurrences of anti-patterns in their WSDLs
[
5
]. Kumar et al. use different object-oriented software metrics
and Support Vector Machines with different type of kernels for
predicting maintainability of services [
6
]. Their experimental
results demonstrate that maintainability of SOC paradigm can
be predicted by application of 11 object-oriented metrics [
6
].
Olatunji et al. develop an extreme learning machine (ELM)
maintainability prediction model for objectoriented software
systems [
7
].
Research Contributions: The main research contribution
of the study presented in this paper is the application of
37 source code metrics (Chidamber and Kemerer, Harry
M. Sneed, Baski & Misra) for predicting 15 Quality of
Service (QoS) or maintainability parameters for web services
by employing Extreme Learning Machines (ELM) using
various kernel methods and two feature selection techniques
(Principal Component Analysis and Rough Set Analysis).
To the best of our knowledge, the research presented in this
paper is the first such in-depth empirical study on publicly
available well-known dataset.
Research Framework: Figure 1 displays our research
framework and methodology. The framework consists of multiple
steps. As shown in Figure 1, we first compute the QoS
parameters for the web services in our dataset. We compute 37
source code metrics belonging to 3 different metrics suite. We
apply two different feature selection methodology (Rough Set
Analysis and Principal Component Analysis) for the purpose
of dimensionality reduction and removing irrelevant features.
We apply Extreme Learning Machines (ELM) with three
different kernel functions (linear, polynomial and RBF). We
create 6 sets of metrics suite, 2 feature selection techniques
and 3 kernel functions and evaluate the performance of all
the combinations resulting in a comprehensive and in-depth
experimental evaluation. Finally, we evaluate the performance
of various models using wide used estimator evaluation
metrics and conduct statistical tests to identify best learning
algorithms.
III. EXPERIMENTAL DATASET
We use a subset of QWS Dataset1 for our experimental
analysis. The QWS Dataset provided by Al-Masri et al. includes a
sets of 2507 Web services and their 9 QWS parameters (such
as response time, availability, throughput, compliance and
latency) which are measured using Web service benchmark
tools [
8
][
9
]. Al-Masri et al. collect the Web services using their
Web Service Crawler Engine (WSCE) and majority of the Web
services are obtained from public sources. We observe that 524
out of 2507 Web Service have their corresponding WSDL file.
Baski et al. present a suite of metrics to evaluate the quality of
the XML web service in terms of its maintainability [
10
]. We
apply the Baski and Misra metrics suite tool on the 524 WSDL
files and obtained successful parsing for 200 files. We use the
metrics proposed by Baski et al. as predictor variables. We
could not include 324 WSDL files as part of our experimental
dataset as we were unable to parse them for computing Baski
and Misra metrics. Hence, we finally use 200 Web services
for the experiments presented in this paper. Redistribution
of the data on the web is not permitted according to the
dataset usage guidelines and hence we provide a list2 of the
200 Web services used in our study so that our research can
be reproduced and replicated for benchmark or comparison.
Figure 2 shows a scatter plot for the number of Java files
for the 200 WSDL files in our dataset. The X-axis represents
the WSDL File ID and the Y-axis represent the number of
Java files. Figure 2 shows that there are several web services
implemented using more than 100 Java files.
700
600
500
s
lsse400
a
C
.fo300
o
N200
100
00
20
40
60
80
100
WSDL ID
120
140
160
180
200
IV. DEPENDENT VARIABLES QOS PARAMETERS
Table I shows the descriptive statistics of 9 QoS
parameters provided by the creators of QWS dataset. The owners
of QWS dataset provide QoS parameter values for all the
2507 web services. However, Table I displays the descriptive
statistics computed by us for the 200 web services used in
our experimental dataset. Table I reveals substantial variation
or dispersion in the parameter values across 200 web services
which shows variability in the quality across services. Sneed
et al. describes a tool supported method for measuring web
service interfaces [
11
]. The extended version of their tool can
be used to compute maintainability, modularity, reusability,
testability, interoperability and conformity of web services. We
calculate these values for the 200 web services in our dataset
and assign them as dependent variables. Table II displays
the descriptive statistics for the QoS parameters calculated
using Sneed’s Tool. Hence, we have a total of 15 dependent
variables.
Chidamber and Kemerer Metrics: We compute several
size and structure software metrics from the bytecode of
the compiled Java files in our experimental dataset using
CKJM extended3 [
12
][
13
]. CKJM extended is an extended
version of tool for calculating Chidamber and Kemerer Java
Metrics and many other metrics such as weighted methods per
class, coupling between object classes, lines of code, measure
of functional abstraction, average method complexity and
McCabe’s Cyclomatic Complexity. We use the WSDL2Java
Axis2 code generator4 which comes built-in with an Eclipse
plug-in to generate Java class files from the 200 WSDL files
in our experimental dataset. We then compile the Java files
3http://gromit.iiar.pwr.wroc.pl/p inf/ckjm/
4https://axis.apache.org/axis2/java/core/tools/eclipse/wsdl2java-plugin.html
to generate the bytecode for computing the size and structure
software metrics using the CKJM extended tool. The minimum
number of Java files are 7 and the maximum is 605. The mean,
median, standard deviation, skewness and kurtosis is 52:39,
45:50, 59:06, 5:43 and 43:94 respectively. Table III displays
the descriptive statistics for 19 size and structure software
metrics computed using CKJM Extended Tool for the 200 Web
services in our dataset. The mean value of AMC as 61:94
means that the mean of the average method size calculated
in terms of the number of Java binary codes in the method
for each class is 62. We compute the standard deviation for
all the 19 metrics to quantify the amount of dispersion and
spread in the values. We observe (refer to Table III) that few
metrics such as DIT, NOC, MFA, CAM, IC and CBM have
low standard deviation which means that the data points are
close to the mean. However, we observe that LCOM, LCO,
AMC and CC have relatively high values of standard deviation
which means that the data points are dispersed over a wider
range of values.
Harry M. Sneed Metrics: Sneed’s tool implements metrics
for quantity, quality and complexity of web service interfaces.
The values of all the metrics are statically computed from a
service interface in WSDL as the suite of metrics is based on
the WSDL schema element occurrences [
11
]. We compute
six interface complexity metrics for all the 200 web services
in our dataset. The six interface complexity metrics are
computed between a scale of 0:0 to 1:0. A value between 0:0
and 0:4 represents low complexity and a value between 0:4
and 0:6 indicates average complexity. A value of more than
0:6 falls in the range of high complexity wherein any value
above 0:8 reveals that there are major issues with the code
design [
4
][
11
]. Table IV shows the minimum, maximum,
mean, median and standard deviation of the size complexity
values for all the web services on our dataset. In addition to
6 interface complexity metrics, we measure 6 more metrics
using the extended version of the tool provided to us by
the author himself: object point, data point, function point,
major, medium and minor rule violations. Table IV displays
the descriptive statistics for the 12 metrics for all the web
services in our dataset.
Baski and Misra Metrics: We compute 6 metrics proposed
by Baski and Misra [
10
]. Their metrics are based on the
analysis of the structure of the exchanged messages described
in WSDL which becomes the basis for measuring the data
complexity. Their metric suitbe is based on WSDL and XSD
schema elements occurrences. Table V reveals the descriptive
statistics of the 6 metrics: Data Weight of a WSDL (DW),
Distinct Message Ratio (DMR), Distinct Message Count (DMC),
Message Entropy (ME), Message Repetition Scale (MRS) and
Operations Per Service (OPS).
We compute the association between 37 metrics consisting
of dependent and independent variables using the Pearson’s
correlations coefficient (r). The coefficient of correlation r
measures the strength and direction of the linear relationship
between two variables. Figure 3 displays our experimental
results on correlation analysis between the 37 metrics. In
Figure 3, a Black circle represents an r value between 0:7
and 1:0 indicating a strong positive linear relationship. A
white circle r rvalue between 0.3 and 0.7 indicate a weak
positive linear relationship. A black square r represents a
value between 1 and 0:7 indicating a strong negative linear
relationship. A white square r represents a value between
0:7 and 0:3 indicating a weak negative linear relationship.
A blank circle represents no linear relationships between the
two variables. For example, based on Figure 3, we infer that
there is a strong positive linear relationship between OPS
and four other variables MRS, OP, DP and FP. On the other
hand, we observe a weak linear relationship between ILC and
IDC as well as ISC and IDC. Figure 3 reveals association
between different suite of metrics and not just associations
between metrics within the same suite. For example, DMR
is part of Baski and Misra metrics suite. DMR has a strong
negative correlation with ISC (Structure Complexity), OP
(Object Point), DP (Data Point), FP (Function Point) and
MERV (Medium Rule Violation) which is part of Harry M.
Sneed metrics suite.
VII. FEATURE EXTRACTION AND SELECTION USING PCA
AND RSA
We investigate the application of Principal Component
Analysis (PCA) and Rough Set Analysis (RSA) as a data
preprocessing step for feature extraction and selection [
14
]. Our
CC
AMC
CBIMC
CAM
MFA
MOA
DAM
LCO
LCNOPMM3
Ce
Ca
LCROFMC
CBO
NDOICT
MWiMRCV
MeRV
MRV
FP
DP
OP
ILC
IDISFCC
IFC
IRC
IDC
MRS
ME
DMR
MDC
ODPWS
objective behind using PCA and RSA is to identify features
which are relevant in-terms of high predictive power and
impact on the dependent variable and filter irrelevant features
which have little or no impact on the classifier accuracy [
14
].
We apply PCA and varimax rotation method on all source
code metrics. The experimental results of PCA analysis is
shown in Table VI. Table VI reveals the relationship between
the original source code metrics and the domain metrics. For
each PC (Principal Components), we provide the eigenvalue,
variance percent, cumulative percent and source code metrics
interpretation (refer to Table VI). In PCA the order of the
eigenvalues from highest to lowest indicates the principal
components in the order of significance. Among all Principal
Components, we select only those which have Eigen value
greater than 1. Our analysis reveals that 9 PCs have Eigen
value greater than 1 (refer to Table VI). Table VI shows the
mapping of each component to the most important metric for
that component. Table VII shows the optimal subset of features
for every dependent variable derived from the original set of
37 source code metrics based features after applying RSA. We
apply the RSA procedure 15 times (one for each dependent
variable). Table VII reveals that it is possible to reduce the
number of features substantially and several features from the
original set are found to be uncorrelated.
Huan et al. mention that Extreme Learning Machines
(ELMs) have shown to outperform computational intelligence
techniques such as Artificial Neural Networks (ANNs) and
Support Vector Machines (SVMs) in-terms of learning speed
and computational scalability [
15
]. ELM has demonstrated
good potential to resolving regression and classification
problems [
15
] and our objective is to investigate if ELMs can
be successfully applied in the domain of web service QoS
prediction using source code metrics. Selection of an
appropriate kernel function depending on the application domain
and dataset is an important and core issue [
16
].
Ding et al. mention that there is a correlation between
the generalization performance and learning performance with
the kernel function [
16
] as in the case of traditional neural
networks [
16
]. Hence, we investigate the performance of the
ELM based classifier using three different kernel functions:
linear, polynomial and RBF. ELMs can be used with different
kernel functions and one can create hybrid kernel functions
also. The most basic, simplest and fastest is the linear
kernel function which is used as a baseline for comparison
with more complicated kernel functions such as polynomial
and RBF. Table VIII shows the performance of the ELM
based classifier with linear kernel function. Table IX shows
the performance of the ELM based predictive model with
second degree polynomial kernel. The polynomial kernel is
more sophisticated than the linear kernel and uses non-linear
equations instead of the linear equations for the purpose of
regression and classification and is expected to result in better
accuracy in comparison to the classifier with linear kernel. The
Radial Basis Function kernel (RBF or Gaussian) is a popular
kernel function and widely used in Support Vector Machine
(SVM) learning algorithm. We use linear kernel to investigate
if the data is linearly separable but also use polynomial and
RBF kernel to examine if our data is not linearly separable
(computing a non-linear decision boundary). We employ four
different performance metrics (MAE, MMRE, RMSE and
rvalue) to study the accuracy of the classifiers. The Mean
Absolute Error (MAE) measures the difference between the
predicted or forecasted value and the actual values (average
of the absolute errors). Table VIII and IX reveals that the
forecast for several predictive model is very accurate as the
MAE value is less than 0:05. For example, the MAE value
for HMS, AM and PCA metrics for predicting Conformity is
0:03. Table VIII and IX reveals that in general the predictive
accuracy for response time, latency, modularity and conformity
is better than the predictive accuracy of other QoS parameters.
Kitchenham et al. mention that Mean Magnitude of Relative
Error (MMRE) is a widely used assessment criterion for
evaluating the predictive accuracy and overall performance
of competing software prediction models and particularly
the software estimation models [
17
]. MMRE computes the
difference between actual and predicted value relative to the
actual value. Table X shows that the MMRE values for ELM
with RBF kernel is between 0:30 to 0:35 for response time,
availability and successability and indicates good estimation
ability of the classifier. Table VIII and Table IX reveals
that the MMRE values for conformity QoS parameter are
as low as 0:05, 0:06, 0:10 and 0:11. Root Mean Square
Error (RMSE) or Root Mean square Deviation
root-meansquare deviation computes the sample standard deviation of
parameters. We do not observe a dominate approach between
the differences between predicted values by the estimator and
PCA and RSA.
actual values. RMSE is also an indicator of the predicted
and observed values. From Table VIII, we infer that the best
RMSE value in-case of ELM with linear kernel is for response
time and latency QoS parameters. The minimum RMSE value
obtained is 0:08 for HMS metrics and conformity parameter
in-case of linear kernel. From Table VIII, we observe that
in-case of polynomial kernel, the performance of PCA based
feature extraction technique is better for some parameters in
comparison to RSA based feature selection technique and
similarly the performance of RSA is better than PCA for some
IX. COMPARING ALGORITHMS USING STATISTICAL
SIGNIFICANCE TESTING
Our objective is to compare several learning algorithms and
assess which algorithm is better. Dietterich et al. review 5
approximate statistical tests for determining whether one learning
algorithm outperforms another on a particular learning task
and dataset [
18
]. We apply the 10-fold cross-validated paired
ttest as described in the paper by Dietterich et al. [
18
]. We have
several combination of subsets of metrics and ELM kernel
functions as learning algorithms. We consider 6 different sets
subset of metrics, a total number of three set (one for each
Metrics (BMS), Metrics derived after executing PCA, and
metrics derived after executing RSA. We consider 6 sets of
performance measure) are used, each with 45 data points (3
kernels multiplied by 15 QoS parameters). Table XI displays
the result of the 10-fold cross-validated paired t-test analysis.
metrics as input to develop a model to predict 15 different
For each of the 3 kernels (Linear, Polynomial and RBF),
QoS parameters. We investigate the application of extreme
6 different subset of metrics are considered as input with
learning machine with three different types of kernel functions:
three different performance parameters. The three different
linear kernel, polynomial kernel, and radial basis function
performance parameters are: Mean Absolute Error (MAE),
with three different performance parameters. Hence, for each
Mean Magnitude of Relative Error (MMRE) and Root Mean
MAE
Linear
Linear 0.000
Polynomial -0.007
RBF 0.021
MAE
Linear
NaN
0.13
0.001
RBF
0.001
0.000
NaN
Mean Difference
MRE
Linear
Linear 0.000
Polynomial -0.005
RBF 0.084
p-value
MRE
Linear
Linear NaN
Polynomial 0.467
RBF 0.000
RMSE
Linear Polynomial
NaN 0.121
0.121 NaN
0.007 0.005
RBF
0.007
0.005
NaN
Squared Error (RMSE). Hence for each kernel a total three set
(one for each performance measure) are used, each with 90
data points (six subsets of metrics multiplied by 15 QoS). The
experimental results of t-test analysis for different performance
parameter (MAE, MMRE and RMSE) and three different ELM
kernels are summarized in Table XII. Table XII contains two
parts. The first part of the table XII shows the mean difference
value and second part shows the p-value between different
pairs. Table XII reveals that there is no significant difference
between the kernel function, due to the fact that p-value is
greater than 0:05. However, by closely examining the value
of mean difference, polynomial kernel yields better result
compared to other kernels function i.e., linear and RBF kernel
functions.
X. CONCLUSION
We develop a predictive model to estimate QoS parameters
of web services using source code (implementing the services)
metrics. We experiment with six different sets of metrics as
input to develop a prediction model. The performance of these
sets of metrics are evaluated using Extreme Learning Machines
(ELM) with various kernel functions such as linear,
polynomial and RBF kernel function. From the correlation analysis
between metrics, we observe that there exists a high correlation
between Object-Oriented metrics and WSDL metrics. From
ttest analysis, we infer that in most of the cases, there is the
difference between the various sets of metrics in terms of the
performance of the estimator is not substantial but moderate.
We observe that the predictive model developed using Harry
M. Sneed (HMS) metrics yields better result compared to other
sets of metrics such as all metrics and Baski and Misra metrics.
From t-test analysis, we can also interpret that difference
between the three kernel functions in-terms of their influence
on the predictive accuracy is moderate. We conclude that none
of the feature selection technique dominate the other and one
feature selection method is better than the other for some
QoS parameters and vice-versa. By assessing the value of
mean difference, we infer that the polynomial kernel for ELM
yields better result compared to other kernels function i.e.,
linear and RBF kernel functions. From performance results,
it is observed that the performance of the predictive model
or estimator varies with the different sets of software metrics,
feature selection technique and the kernel functions. Finally,
we conclude that it is possible to estimate the QoS parameters
using ELM and source code metrics.