Web service monitoring tools have become an essential component for Service Level Agreement (SLA) because they can collect real quality values of quality attributes to estimate the quality level of services. Since users monitor more than one quality attribute at the same time, quality levels are prone to vary during monitoring time, producing a con ict among quality attributes. In this study, we conduct an empirical study in order to identify potential con icts among quality attributes during Web services monitoring. For this purpose, we monitor a TravelAgent service in two scenarios: 1) monitoring attributes in isolation, and 2) monitoring attributes in pairs. Bootstrapping is used to estimate the quality level for each scenario, then we compared the quality levels in order to identify degradation. The results have shown that Response Time and Accuracy are the most con icting attributes during monitoring.
Quality of Service (QoS) is a major issue in Web Services because services are
ussually third-party software integrated in large systems. QoS is a set of
quality attrbutes (e.g. availability and performance). The QoS analysis has become
a crucial task for service users, since they do not know the real quality level
provided for Web services. For this task, monitoring tools are used to collect
their quality values and evaluate the quality levels of Web services. Monitoring
tools collect quality values by means of metrics de ned for quality attributes,
and applying two monitoring strategies [
The paper is organized as follows: Section 2 and 3 de ne our research problem and related work, respectively. Section 4 exposes the planning and execution of our evaluation. The results and analysis are in Sections 5. Threats to validity and Conclusions are presented in Section 6. and 7. 2
Today monitoring tools are part of Service Oriented Architecture (SOA) infrastructure. In general, they can be set up on the service side, on the client side or as \man in the middle". Since quality attributes have di erent metrics to measure their quality levels, a di erent monitor should be used for each attribute separately. Thus, monitors can use di erent methods to collect quality values depending on the quality attribute they monitor. However, these methods can con ict between them producing a degradation in the quality level for one or more quality attributes. In order to identify these potential con icts, we propose an empirical study to evaluate potential con icting quality attributes during Web service monitoring from a viewpoint of their users. 3
Mairiza et al. [
On the other hand, Zheng et al. [
Mairiza et al. [
In this section, we present the objective and hypothesis of our empirical study, as well as the independent and dependent variables, subjects and experiment planning. 4.1
The objective of our empirical study is de ned by following the analysis model
GQM (Goal-Question-Metric) [
Where Qlevel(S; Ai) represents the quality level of the attribute Ai monitored in isolation, and QlAejvel(S; Ai) represents the quality level of the attribute attribute Ai monitored along with the attribute Aj .
Accuracy
Availability
Monitoring Tool: Currently there are many monitoring tools reported in the
scienti c community, such as Cremona [
Quality Level: Firstly, we de ne formally quality value. A quality value is a numeric value, which is the result of applying a metric of a quality attribute in a Web service in an instance of time. We present a quality value as: Qvalue(S; Ai; Mij ; t) = v (1) where v 2 R is the quality value of the quality attribute Ai with i = 1; :::; n using the metric Mij with j = 1; :::; m and n; m 2 N+ in the time t in the Web service S.
The quality level for a quality attribute is composed of a set of quality values collected in an interval of time. Since quality values present smooth variations in the time, quality levels are represented by a range of quality values within a con dence factor. For example, the response time is in the range [15 ms, 45 ms] for 95% of the cases. The quality level is de ned by:
Qlevel(S; Ai; Mij ; t1; t2) = [vmin; vmax] for C% of the cases (2) where t1 and t2 are the interval of time (start and end time, respectively) collecting quality values, vmin and vmax are the minimum and maximum quality values, and C is the con dence factor of the quality level.
Additionally, we de ne three types of degradation comparing two samples
of the same quality attribute collected in a di erent period of time: Absolute
degradation, when there is not an intersection in the quality level of the two
samples. Relative degradation, when there is an intersection in the quality level
of the two samples. No degradation, when the quality level of the two samples is
the same. We also de ne three types of con icts between two quality attributes
based on Mairiza et. al [
The subject of our study is a TravelAgent service. This service look for hotel rooms, airline tickets and renting cars. TravelAgent is a service orchestration composed by three third-party services: CarReservarion, FlightReservation, and HotelReservation. In order to reproduce a real scenario, Web services were allocated in di erent locations and hosted in di erent operating systems. It was rented four virtual machines in Google Cloud Platform3 (Table 3). TravelAgent service was installed in Apache ODE 4 and deployed in Apache Tomcat5. All third-party services were developed in JAVA and deployed in Apache Tomcat. Web service Location SO SO Distribution Number Cores Processor
Our experiment was designed into two steps: Step 1: Creation of Di erent Monitors. An independent monitor was created for each quality attribute according to Table 2 using FlexMonitorWS. Table 4 shows the monitoring pro les for each quality attribute. 3 https://cloud.google.com/ 4 Orchestation Director Engine which execute business process de ned in WS-BPEL. 5 A open source web server and servlet container developed by Apache Software Foundation. Step 2: Execution. In order to identify degradation in the quality levels, monitoring was executed during 24 hours considering two scenarios: 1. Scenario 1: Monitoring in Isolation. Every quality attribute is monitored when no other attribute is monitored. The aim of this scenario is to create a basis state of the quality level of the Web service (Q1). 2. Scenario 2: Monitoring in Pairs. All possible pairs of quality attributes are monitored at the same time in the same Web service (Q2). The aim of this scenario is to produce a degradation in the quality level of at least one attribute (Q3). 5
Since the quality level is prone to vary in the time, we estimated the magnitude
of these variations and its error range by applying bootstrapping. It is a
statistical technique that estimates the sampling distribution by making random
re-sampling, with replacement, from the original sample [
Most of the observed degradations were caused by the number of invalid
responses returned by the service. Invalid responses were caused by lack of memory
6 All collected quality values are in: http://www.students.ic.unicamp.br/
~ra153621/empirical-conflicts-qos-monitoring.html.
1.0
0.8
0.6
F
D
C
0.4
Threats to conclusion validity concerning the relationship between treatment
and outcome. An incorrect execution of the experiment and incorrect treatment
of the collected information can produce biases in the results. In order to
minimize this threat, the experiment was designed in order to be reproducible by
someone else, following the guidelines purposed by Wohlin [
Threats to external validity concern the possibility of generalizing our results. The generalization of our results depends on to take a representative sample of 1.0 0.8 0.6 F D C 0.4 0.6 F D C 0.4 0.2 monitoring quality values. The Web service was monitored for 24 consecutive hours, and generalize its quality behavior by using bootstrapping. However, this is limited by the subject evaluation because it was only for a single service. 7
This work has presented an experimental study for Web services monitoring, in order to identify potential con icts between quality attributes. For this aim, a set of quality attributes were monitored during 24 hours considering two scenarios: monitoring quality attributes in isolation and in pairs. The results showed that 1) Response Timeand Accuracy were the most con icting attributes since it presented quality degradation during monitoring with all the other attributes. 2) Fault injection used to monitor Robustness was the most intrusive technique because faults are sent to the services in order to produce errors in the service, and subjecting the service under stress condition or disabling it. 3) Active monitoring can become intrusive in the Web service generating degradation in the quality of service, since this strategy send request directly to services. New studies are necessary in order to evaluate more complex services and generalize the result for Web services.