The errors in BPEL processes that are only detected at runtime are expensive to fix. Several modelers and process engines for BPEL exist, and the standard defines basic static analysis (SA) rules as a detection mechanism for invalid processes, but the actual conformance of BPEL modelers and engines regarding these rules is unknown. We propose to develop test cases to evaluate the conformance of BPEL modelers and engines regarding static analysis. The evaluation results enable decision makers to identify and use the most conformant engine and modeler that detect errors before runtime and therefore reduce costs.
BPEL, a standard [
Each static analysis rule defines constraints for at least one BPEL element, e.g., rule #47 enforces, among other conditions, that any received non-empty message must be stored in variables. For example, consider a developer implements a simple BPEL process2 with two variables (input and output variable) which awaits a message (onMessage) which instantiates the process, copies the contents of the 1 The rules are enumerated from 1 to 95, but 49 is missing, thus 94 rules exist. 2 The process is available at https://lspi.wiai.uni-bamberg.de/svn/betsy/ sa47-test.zip - Accessed 02/14/2014 input variable into the output variable (assign) and returns the output variable (reply), but forgets to store the received message in the designated input variable, hence violating rule #47. Against expectations, the error is neither detected by the two widely used Open Source BPEL IDEs, Eclipse BPEL Designer v1.0.3 and the OpenESB IDE v2.3.1, nor by the BPEL analysis tool BPEL2oWFN. Moreover, only the BPEL engines Apache ODE 1.3.6 and bpel-g v5.3 correctly reject the erroneous process whereas OpenESB v2.3.1 and Orchestra 4.9 falsely accept and deploy the process. Upon execution of the process, both engines show behavior which is hard to debug: OpenESB returns null with the error of a NullPointerException and Orchestra returns a timeout with no error trace. In this particular case, none of the modelers or analysis tools and only two out of four engines were able to detect this basic error.
Thus, this preliminary evaluation raises the following research question: What is the conformance to the static analysis rules of BPEL modelers and engines and why is this the case? 2
Static analysis regarding BPEL has been studied extensively in literature. Each
considered approach was analyzed by investigating three aspects: the BPEL
version, the number of test cases, and the amount of static analysis rules covered
by the tests. The approaches [
Whereas the discussed approaches check the standard conformance of BPEL
processes, none of them evaluates the standard conformance of BPEL modelers
or engines. Harrer et al. [
To answer the research question, we aim to a) create test cases (d−−er−i→ve in Fig. 1) for the 94 static analysis rules and b) use these test cases to analyze static analysis conformance of BPEL modelers and engines (e−v−a−l−u−a→te in Fig. 1) with betsy, i.e., determining the degree of static analysis conformance.
The derivation of the test cases for each static analysis rule is subdivided into six steps. First, the related elements and attributes are extracted from the textual representation of the rule and partitioned into groups. Second, the elements and attributes are permutated into a list of possible combinations. Third, we identify valid, invalid, and meaningless combinations according to the standard restrictions. Fourth, we calculate the distance from each invalid combination to every valid combination and pair up the least distant combinations. Our distance metric is the amount added and removed elements and attributes. Fifth, to create the test case we select the most feature-poor process from a test set of valid processes (e.g. the betsy test set) that fits the valid combination. Sixth, we mutate the valid process to an invalid one corresponding to the invalid combination. This approach increases the quality of the tests as it ensures that the erroneous process solely violates a single error condition. Especially the first and the third step are hard to automate as interpreting prose is nontrivial. Thus, four steps are automated whereas the other two are done manually. Following our running example, we applied the first step of our proposed procedure onto rule #47 and determined the formalization of influencing factors in the listing below. Step two to five are not shown. Regarding step six, the test case described in Sect. 1 implements the error condition represented by the permutation marked as bold.
{empty message, non-empty message} × {invoke, receive, reply, onMessage, onEvent}×
{incoming, outgoing}× {variable assignment, no variable assignment}×
{part assignment, no part assignment}
An engine passes such a test case if the process is rejected during deployment. But there may be false positives, i.e., the process is rejected by an engine because of an unsupported feature or an internal error. To prevent misleading results, we propose to take the betsy conformance evaluation into account. betsy reveals [4, p. 6] that full standard conformance is far from given by detailing which BPEL feature is supported by which engine. To avoid false positives during the evaluation of a single error type, we suggest to use a pair of BPEL processes, a fully functional and an erroneous one. We assume that if the engine rejects the valid process, it is not able to detect this error type. But this assumption introduces false negatives, as the engine may reject the erroneous process by static analysis and the valid one by missing feature support. To counter this, we propose to evaluate the log files for any hints on why the deployment failed. The evaluation of the BPEL modelers is analogous.
The evaluation with test pairs of valid and invalid tests reveal the quality of the error detection, making the engines and modelers comparable in this regard. As the degree of error detection has an impact on the development costs, this metric may be leveraged for buying decisions. In addition, it can also be used for improving the products and act as a regression test suite.
The requirements to use our approach are 1) a process language standard
defining rules for valid processes and 2) a feature complete set of valid processes.
We use the process language BPEL in this case study, but the approach is also
applicable for other process languages, e.g., for the Business Process Modeling
and Notation (BPMN) [