Testing executable business processes, e.g., developed in BPMN 2.0 or WS-BPEL, is an important task within development projects. However, this task is labor-intensive due to the amount and scale of interactions of a test framework (e.g., BPELUnit) with the process under test. We want to help testers to debug test case failures by visualizing the test run information as a BPMN diagram. We developed a visual, BPMN-based notation for visualizing test runs, implemented a generation from BPELUnit test suites and execution logs to the newly defined format, and applied this in an industrial project. While no thorough validation has yet been performed, early results indicate better understandability of our notation compared to raw test logs.
Testing executable business processes is an important task in many digitization projects: errors in
critical business processes are a huge risk for reputation and income of organizations. Therefore,
unit testing, e.g., with BPELUnit [
However, analyzing failing unit tests in executable business processes is often dificult,
because they integrate many partners that potentially are performing activities in parallel. As
such, test case logs containing successfully executed and failed activities alongside passed input
and output messages become large and are hard to analyze. While test frameworks indicate
which activity failed, further analysis is often required to pinpoint the underlying problem. This
is especially true, when test cases are generated, e.g, by using combinatorial test design [
We developed an approach to generate BPMN models, which help developers and testers to
easier analyze failed test cases. It builds upon a test case subset for BPMN [
This paper starts with a presentation of related work (section 2). Afterwards, the generation approach and underlying meta models are presented in section 3, which are implemented in a prototype application (section 4). At the end, a small validation in an industry project is presented in section 5. Finally, conclusions are drawn and outlook on future work is made (section 6).
Visualizing test cases has been a research area of interest for quite some time. For example,
Cornelissen et al. [
But visualization is not only concerned with the test execution per se but also with its results.
For example, Opmanis et al. [
However, more similar to the goals of this research, studies have been concerned with
visualizations to better detect faults: Jones et al. [
Existing tooling can also be used to present more data to testers. For example, there is a tool for the Camunda BPMN engine 1 to visualize test coverage metrics as a graphical overlay over the process.
A BPMN Test Execution Visualization is a BPMN model that shows all activities executed or
supposed to be executed during a test run, which conforms to the following constraints (derived
from the constraints for model-driven testing of executable business processes [
If a mock sends a one-way message, a message throw event is added to its pool, if it receives a one-way message, a message catch event is added. If it initiates a two-way message exchange, a service task is added, and if it waits for a two-way message exchange a message catch event is added followed by a message throw event. If the mock waits for a specified duration, a timer event is added, and parallel branches within a partner is represented with parallel gateways. Because BPELUnit is block-structured, it is easy to also add the merging parallel gateway.
Within this section we shortly describe some design considerations for the prototype application, which contains as much functionality as required for trying it in the industry project (see section 5). The prototype is available at https://github.com/dluebke/bpelunit-viz.
Because we want to use colors for denoting tasks’ error status, we were required to settle on one tool, because colors are not standardized in the diagram interchange format of the BPMN 2.0 specification. We settled on the Camunda Modeler because it allows us to show the documentation in a comparatively large text area in a side pane. Thus, developers can easily select BPMN elements of interest and directly see exchanged information.
We used Java’s DOM and XPath API to read elements and attribute values from the BPELUnit log. We have planned to add the processing of the BPELUnit test suite itself as described above. However, the prototype does not yet resolve the original operation name as foreseen in the concept.
Internally, all metamodels are represented by model classes (BPELUnit, BPELUnit Log, BPMN 2.0). BPMN 2.0 can be serialized via custom serializers for the model classes.
The resulting layout of a test case visualizaton is illustrated in Figure 2.
Because developers want to use the generated BPMN diagrams without further efort, we were required to layout the diagrams as well. We simply layouted all flow nodes in a pool horizontally. This approach works reasonably well because the diagrams contain no loops, few gateways, and are mostly sequential.
To reduce the average length of message-flows and number of line-crossings and thereby making the diagram easier to comprehend, we ordered the pools based on the number of contained flow nodes (events, tasks and gateways): The pool representing the process under test is placed in the middle. The pool containing the most flow nodes is placed on top of it, the pool with second most flow nodes is placed below it. The ordering progresses by adding the pools descendingly sorted by the number of contained flow nodes on top and below already layouted pools.
Some tuning was required due to the the large amount of message-flows and pools: Because nearly all task had straight vertical message flows to the process-under-test pool, most message lfows overlapped. We resolved this problem by slightly ofsetting the horizontal starting point within each pool.
For addressing quality concerns, Terravis [
The project has so far used the visualization for 25 of its processes that are covered by CTD tests executed with BPELUnit. The largest BPELUnit log file is 48 MB large containing 328 test cases. The log file size is mainly influenced by the (large) XML messages being exchanged in the covered process. The visualization thus reduces the cognitive load required by developers because it filters out payload information initially although allowing users to access it via the elements’ documentation as shown in Figure 4.
Feedback by developers was positive and they indicated that the visualization helped them to diagnose problems. However, further validation is required.
In this paper we presented a way to visualize test cases for executable business processes and their executions as BPMN models. The proposed algorithm has been implemented in a tool and successfully applied in an industrial case. Consequently, BPMN can be used as a single language to model the business processes, model or visualize test cases (depending on the testing approaches), and visualize test execution results.
In future work we like to further validate the approach by conducting surveys and experiments for comparing log-based analysis with analysis performed using our models with regard to efectiveness of finding and fixing problems in test cases and executable business processes.
We also made the prototype implementation publicly available so that everyone can use it and perform such studies with us or independently. We are happy to collaborate with others concerning research into optimizing the testing of executable business processes!
We like to thank all team members of the industrial case project Terravis for their participation.