Service-oriented architectures have found their way into industry to enable better business-to-business cooperations. With this software architecture new challenges for software development and testing appeared. In this proposal we discuss the problem of testing these complex, and distributed systems in dedicated test environments. We argue that state and con guration of the production system can in uence system behavior in an unexpected way, and that test environments do not re ect the nal system adequately. Therefore we propose the development of an online testing method to detect state-based faults, and discuss related research challenges and solutions.
A service-oriented architecture (SOA) is a software architecture that aims at
reusability and interoperability by exposing functionality as loosely coupled
services, that can be composed to form business processes. Testing such systems
is aggravated by stakeholder separation: third party services are black-boxes
for integration testers, and in uence in their evolution and maintenance is not
permitted. Dynamic features, like ultra-late binding or dynamic composition of
services, allow a exible and dynamic way of composing a concrete system just
at runtime, but restrict the ability to test a priori. The context in which a service
will be used is often unknown at the service's development time, imposing
problems for service testers to predict and foresee possible requirements and usage
scenarios. The limited testability of service-oriented systems imposes the need
of reconsidering and redesigning traditional, and inventing new testing methods
and processes, as mentioned in [
On the other hand, high availability requirements force business critical systems to be continuously available, and direct them to evolve at runtime as new requirements come up. The system cannot be shutdown for the deployment process, and erroneous behavior due to software recon guration has to be detected while the system is operational, i.e. online. State comprises in our context information about previous executions, the concrete set of installed and active programs, as well as history of preceding con guration activities.
This proposal suggests the development of a method to identify state-based faults in online recon guration of service-oriented systems. Recon guration includes evolution of business processes, services, business logic and changes in the execution environments.
The proposal is structured as following: First, related work on SOA testing is summarized. Section 3 states the problem and formulates the research hypothesis. The related research challenges and possible solutions are discussed subsequently. The research plan is outlined in Section 5, followed by the evaluation plan in Section 6. The expected outcomes are exposed in Section 7. 2
In uenced by the need of new testing methods and techniques for SOA, many
approaches on unit testing of atomic or composite services, integration and
interoperability, as well as regression testing can be found in literature [
Bucchiarone et al. discuss in [
Research in service-oriented testing has also looked at the eld of online
testing, such as by Bertolino et al. for interoperability testing [
Tsai et al. propose in [
All approaches abstract from the complex and heterogeneous environments the services are operated in, and do not mention their in uence on systems' behavior. 3
Testing of SOA systems means testing of organization-spanning systems-ofsystems. Setting up an accurate test environment, that represents the nal environments of all involved parties in which services are used is in most cases unfeasible. Many parts of third parties have to be stubbed or mocked, even if there is a lack in provided information. Di erent and heterogeneous execution environments and their con guration have to be recreated, whereas a clean initial system state still does not re ect the production environment. Beside resource limitations, also time restrictions hinder adequate set-ups.
Complete recreation of the execution environments and their con guration for the test environments involves high costs, and even if done carefully, is often not accurate, leading to the risk that even tested software fails during online integration in the production environment.
We want to determine how the state of the services, of the business logic located behind and of the execution environments can in uence the system behavior in an unexpected way. Our testing focuses on nding faults that deal with incorrect state transitions of the con guration of the connected services.
To conclude, state-based faults cannot easily be revealed during testing of complex, large, distributed SOA systems in dedicated test environments. A method for identifying state-based faults during online recon guration of serviceoriented architectures is needed.
On account of this the research hypothesis is: \Online testing is an e ective strategy for detecting state-based faults in service-oriented systems." 4
An e cient online testing method to detect recon guration faults has to address following challenges.
What are typical recon guration faults in SOAs, and which state and con guration produce those, even if software and services have been o ine tested?
The main challenge is to determine which state and con guration data in uence program execution in a SOA environment. This has to be narrowed down by the question: \Which information is often left out in the set-up of test environments?".
To de ne possible faults it has to be clear how recon gurations of servicecentric systems are propagated to local and remote systems. Possible faults have to be partitioned in those that can easily be detected during o ine testing, and those that cannot. Latter faults have to be subdivided again, based on the question: \Which errors are already handled by the middleware, and which have to be handled by services and systems themselves?"
Regarding state and con guration, security and authentication policies are often not known or accurately implemented in a test environment. Many errors, like service unavailability and deployment errors, will be detected and partly solved by the middleware. During composition, and service execution, typical integration faults, like interface, data format, and communication protocol mismatches are to be expected. Further, dependency errors caused by evolution of parts of the system can occur, especially if some of the services and systems involved are stubbed or mocked in the test environment. It is important to analyze the impact software evolution has on the existing systems, either local or remote, and which layers of a SOA are a ected. This can reach from changes in the business process layer, to the business logic layer and also to changes in the execution environment. Changes in the backend of a service, be it the business logic or the con guration of the execution environment, can have rippling e ects on local or even remote systems. These changes can be for instance, policies, new installed software, changes of database schema or changes of the database management system. Such modi cations can cause service level agreement violations, because the network load increases, the new database management system responses slower, or access is now denied.
How should an e ective online testing framework for SOA integration and system testing be designed, and what are the requirements for such a framework?
Challenging during design and development of an online testing method, are test isolation, performance, and application enhancement e ort. If a system's behavior is assessed online, it has to be guaranteed that testing is isolated from the production system, and no unintended side e ects appear. Test execution can only take place if no performance decreases or even unavailability of service are experienced. Enhancement costs for applying online testability mechanisms have to correspond to the positive impact accompanied with them. This implies that the tradeo of online testing is corporeal. But how can the e ectiveness of an online testing method be assessed and measured? Evaluation could be based on the capability of online testing to reveal faults in contrast to o ine testing.
Information regarding the state of execution environments and infrastructures (e.g. application server, messaging bus, etc.) should support online testing to reveal faults. \But what is an accessible and uniformed way to provide state information to testing entities?"
Monitoring capabilities can be used to notify about recon guration changes in the system. An observer service, installed on each server involved in the SOA environment, could provide access to this information to interested and authenticated testing entities.
How can test cases, oracles, and system models be derived or generated?
In many SOA environments, abstract business models, business process models or UML documents, as well as service descriptions are present. Those represent a good foundation to derive system models, test scenarios and cases.
One technique would consider the running system to be correct and use it as test oracle. The new, adapted version of the system is tested, and the test output is compared with the output of the old, stable system. Capture and replay techniques can help to increase observability, and controllability of the testing process, because old input and output sequences are recorded and reused ad libitum. 5
To address the rst research challenge, asking what are typical recon guration faults in SOA-based systems, we have implemented an online testing method providing test isolation in a case study. We could gain a rst insight in state-based faults, mainly caused by missing required packages, or wrong class bindings. To develop a method for identifying state-based faults in online recon guration, we have to understand better which state and con gurations are relevant in SOA systems, and how these in uence the testing e ectiveness and accuracy. In the coming year, we plan to set up a SOA laboratory in order to examine con guration and state information signi cant during system recon guration. Based on the results we want to develop a fault taxonomy for faults caused by evolutions in distributed, heterogeneous runtime environments.
In the next year, our online testing method has to be enhanced to identify these faults, and to be applicable in industrial SOA environments (e.g., IBM product suite).
Subsequently, we will explore capabilities to reduce the application enhancement e ort. This will include automatic generation of test cases, and of testability artifacts allowing runtime service assessment. With this outcome, the online testing method will be extended to an open source online testing framework, providing support for test generation and execution.
Throughout the research, the outcomes will be evaluated based on industrial case studies. 6
The SOA laboratory is a network of distributed servers, on which services, installed on di erent execution environments, form actual applications, inspired by industry. Communication is handled by a centralized service registry and bus.
This laboratory will be used to conduct representative case studies, following
Yin [
In collaboration with our industrial partner, we also want to assess the e ort to prepare an application, with support of our framework, for online testing. For that, we will measure, for example, the required time and the number of necessary changes in the existing system. A comparison of e ort and fault nding capabilities should indicate if online testing has a positive tradeo . 7
The resulting contributions of the doctoral studies include: 1. A new approach for online testing to detect state-based faults. 2. A series of fault models classifying faults that appear especially in the production environments. 3. Empirical evidence concerning the e ectiveness of the proposed approach. 4. An open source tool for online test support, including generation and execution of test cases.
Our results should corroborate the hypothesis by indicating the e ectiveness of online testing to detect state-based faults during system recon guration.