-Increasing automation of business processes and industrial demand for complex workflow features have led to the development of more flexible and agile workflow concepts. One of those concepts is the use of goal-oriented workflows, which rely on ideas derived from agent technology like describing the workflows based on a goal hierarchy. While this reduces the gap between business view and IT view and allows for easy implementation of contengencies, the concepts have greater conceptual abstraction obscuring the control flow and reducing the ability of workflow engineers to identify specification flaws in the workflow. This paper shows an approach to address this problem by presenting a system for testing and validating workflows within a specified parameter space. The system allows the definition of test cases (scenarios), each of which contains parameter states applied during workflow execution. The workflow engineer can define a set of scenarios for a workflow testing specific situation that are likely to occur during operation or are otherwise interesting corner cases, allowing automated tests and correction of faults before deployment of the workflow in production environments.
Business process management (BPM) is considered a very
promising strategy that helps in aligning companies towards
effective and efficient business operation [
One fundamental problem with existing solutions that has
been experienced in practice is that modelling notations like
event process chains (EPCs) or the business process modelling
notation (BPMN) are activity oriented and thus focus heavily
on the ordering and conditions of activity execution. This leads
to a particular neglect of the underlying process motivations,
which are only implicitly existing during workflow
elicitation. Without this so called workflow context perspective [
In order to equip a workflow modeller with a toolset to better understand the possible runtime behaviour of more abstract workflows, simulation and validation gain importance. In this paper a new concept and implementation for a simulation based validation approach of workflows is presented. It allows to specifiy execution scenarios and automatically evaluate them with respect to expected process outcomes. The approach does not depend on the concrete workflow notation in which processes are described but only assumes that specific workflow management facilities are available. Thus, the approach is viable for validating traditional e.g. BPMN based workflows as well.
The rest of the paper is structured as follows. In Section II related work with respect to verification and validation approaches of workflows is discussed. Thereafter, in Section III the concept of the new simulation based validation approach is presented. Its implementation and usage is explained in Section IV and the usefulness of the approach is further illustrated by an example application taken from our industry cooperation partner Daimler AG. The paper concludes with a summary and a short outlook on possible future enhancements.
Literature and practical application contains a large variety of validation and verification approaches for workflows. They can roughly be divided into two categories. The first category consists of formal static analysis of the workflow model to identify conceptual or implementation flaws. The second category comprises of validation using simulation-based execution of the workflows.
Formal verification approaches of workflows apply a
number of techniques, such as propositional logic [
Furthermore, verification approaches are often quite limited on what they can guarantee. For example, while there are excellent approaches to guarantee correctness of the workflow diagram like ensuring that branches in the workflow are terminated with a correct join element, they are generally unable to verify non-trivial semantics like task instructions written in a programming language or complex runtime behavior.
This problem can be approached by the validation
techniques in the second category. Instead of statically analyzing
the workflow models, the workflow is executed in a simulated
environment which attempts to imitate the environment in
which the workflow is planned to be deployed. Examples of
this approach include a variety of tools like LSIM [
In contrast, the focus of our approach has primarily been the validation of the workflow using predefined test cases centering around expected real world scenarios. The disadvantage of this approach compared to verification is that the correctness can only be ensured within the given conditions of the test. Since the number of configurations in any nontrivial workflow is very large, exhaustive search becomes infeasible. This means the approach can only validate the workflow to a certain degree and may not detect all errors present. The advantage of the approach is that the complexity of the workflow language is irrelevant to the test as long as there is a workflow engine capable of executing the workflow. Furthermore, the current approach uses few assumptions about the workflow itself allowing the system to be useful for different kinds of workflow notations. While the focus has been on validation of workflows, the approach can be extended to include simulation of the environment in which the workflow is running but which is, by itself, not part of the workflow. This includes logistic operations, production machinery, behavior of workflow participants and market influences. This allows additional applications beyond workflow validation like workflow optimization similar to the tools mentioned above.
In the following we will present this simulation-based testing approach which uses simulation of workflow participants to validate the correctness of workflows. The focus will be on goal-oriented and agile workflows, however, as mentioned, the approach itself can at least partially be applied to any workflow which relies on interaction with participants.
Due to its application in industry and business automation
and intense research interest, a great variety of workflow
approaches and notations are available. While our validation
approach is designed to be generic, two kinds of notations
in particular were the focus of the project. The first is the
well-known Business Process Modeling Notation (BPMN, see
[
In addition, GPMN workflows contain a context which represents the current workflow state. This context is used as the belief base of the converted BDI agent at runtime. This context may be changed during execution of the workflow, either directly by the workflow itself or by effects outside the workflow. Changes in the context can directly affect the workflow and thus the agent behavior by influencing adoption, persuit and rejection of goals.
Both GPMN and BPMN offer language features which allow the workflow engineer to implement different execution paths or branches. In the case of BPMN workflows, this is accomplished using the gateway element, which can split the control flow into either multiple paths executed concurrently or diverts the flow towards one of multiple possible control flow edges. The implementation of execution paths in GPMN workflows is more subtle and indirect. Depending on the state of the workflow context, different goals may become active resulting in the execution of different BPMN plans. This control flow subtlety of implicit control flow paths in GPMN workflows increases the difficulties of a workflow engineer to accurately predict possible runtime execution paths and is the primary motivation for our validation approach.
The core idea of our approach towards validating such workflows is the use of automated tests. The goal of the approach is to provide the workflow engineer with tools to specify realistic business cases which are likely to occur in the real world which are then applied to a workflow instance.
As a result, the test coverage of the approach is limited to those cases, however, it is aimed at the most likely situations, providing assurance of validity in the cases most likely to occur after deployment. Since only the most trivial workflows workflow management system, thus allowing the workflow can be automatically executed merely using a workflow engine to continue executing. In order to simulate this behavior, the and since most workflows require interaction with workflow workflow client application used by the workflow participant participants or automated systems while running, additional needs to be replaced with an automated workflow client system components are necessary beyond the workflow engine application which simulates its behavior and the behavior of itself. Moreover, while most workflows specify a range of the workflow participant. possible responses by workflow participants, they generally do not specify which responses will influence workflow behavior, C. Client Application thus necessitating the specification of additional information by the workflow engineer before the test.
Consequently from a simulation perspective, a workflow engine executing a workflow is not a viable simulation model for validation workflows since it lacks sufficient detail even for simple automated execution, much less being a realistic representation of a production environment using workflows.
Therefore it is necessary to add additional components to the system which are equivalent or at least sufficiently similar to their production counterparts to represent an adequate model of a workflow in production use (see Figure 1).
One component which is routinely part of workflow systems in businesses is a workflow management system (WfMS). The task of a WfMS among other things is to facilitate interaction between workflows and workflow participants. This is generally done by providing work items, which are packages generated by the WfMS on behalf of the workflow containing all the information needed by the workflow participant to accomplish their part of the workflow. The WfMS then distributes the work items among the workflow participants using a variety of approaches such as roles. As a result, the simulation model of a realistic test system needs to include a WfMS which accurately represents a WfMS used in production.
Work items generated by the WfMS not only include information for the workflow participant but often ask the participant to gather and provide external information like customer data or processed documents for the workflow. This is defined in the workflow with the specification of typed parameters in tasks which generate work items. This information often influence further behavior of the workflow at critical junctions like BPMN gateways or goal deliberation. Since real workflow participants are not available during test runs, it is necessary to simulate their actions, including the supply of external information.
Work items are generally retrieved and processed by the workflow participant using a workflow application client interacting with the WfMS. The work items are retrieved, processed by the workflow participant and finally comitted back to the The simulated workflow application client is required to provide the information which is normally provided by the workflow participant. As a first step, the client identifies the workflow tasks which generate work items and require the workflow participant to provide information in the form of work item parameters by examining the workflow model and the models of possible subprocess, such as BPMN workflow fragments in case of GPMN workflows. Parameters are typed and thus already have a limited parameter space. However, this parameter space in cases such as string types is extremely large, precluding an exhaustive test of the full parameter space. Since a complete verification of the process using this approach would also require to test the cartesian product of the parameter space of all parameters provided by the workflow participant, the complexity of such a verification exceeds the limits for a practical test and cannot be considered a useful approach.
As a result, it is necessary to restrict the scope of the test to only include the part of the parameter space which includes the most promising cases, such as corner cases of branching decisions and validating the workflow only for those cases. Since the test cases cannot be identified automatically, the workflow engineer has to define the parameter space that needs to be tested. This is accomplished by the system by allowing the workflow engineer to define test scenarios. Scenarios represent a subset of the full parameter space of the workflow participant interaction with the workflow. For each parameter in the process the workflow engineer can define a set of parameter values which are used to process work items while the workflow is executing. If the workflow engineer defines multiple values for each parameter within the same scenario, the cartesian product of those values is tested at runtime.
The workflow engineer can define multiple scenarios for each workflow. When the test is started, the first scenario is selected and the workflow is started repeatedly, once for every element of the cartesian product of the parameter values in that scenario. Once the scenario finishes, the next scenario is selected until all scenarios have been tested. An event log is kept during each execution, recording notable events occuring at runtime for later analysis. The workflow engineer can use this log to identify errors in the workflow and correct them before the workflow is used in a production system. In addition, a test report is generated and can be reviewed.
Errors in the workflow can be unrecoverable states of the workflow like a raised exception during execution or unintended behavior of the workflow such as performing a faulty execution order of tasks. In additions, it is also considered to be an error if the workflow returns the wrong results or reaches the wrong internal state. Since the workflow engineer has to be informed about such errors occuring, monitoring of the workflow is required. On raised exceptions, executed steps of the workflow and state changes the workflow engine can generate a workflow event which is passed through the WfMS to the client application for review by the workflow engineer. It would also be desirable to allow the workflow engineer to define a validation function which receives the information provided to the workflow and the final state and result of the workflow.
The following section will elaborate on the individual parts of the testing system. It will include an overview of the workflow management system and provide details about the simulated workflow client application and how the workflow engineer can define the parameter space subset for each scenario.
As mentioned in the previous section, the testing system requires a minimum of three components. In order to execute the workflows themselves, a workflow engine is needed, a workflow management system is needed for work item management and user interaction and there needs to be a special workflow client which simulates user behavior.
Since GPMN workflows are translated into BDI agents with
BPMN plans and thus require a workflow engine which can
execute both, the Jadex Active Components Platform [
The second required component is the workflow
management system. This system component is implement largely
based on the reference model of the Workflow Management
Coalition [
The services of the WfMS are divided into internal and external services, the former implementing the actual functionality of the WfMS while the latter, represented by the agents, act as an interface which can be used by workflow client applications to connect with the WfMS. The Jadex platform itself represents the workflow engine and enactment service of the WfMS by providing the necessary support for instantiation of workflow models.
Workflow models are managed by the WfMS using a process model repository service. This service supports the addition and removal of process models by employing the Jadex library service which allows Jadex to dynamically load new models, resources and executable code by linking directories or jar-archives. In addition, it offers access to workflow models for workflow client software, which is a necessity for the testing system since the simulated workflow client application requires the workflow model in order to identify tasks in the workflow which require interaction with a workflow participant.
The Authentication, Access Control and Accounting (AAA) Service of the WfMS provides additional workflow services like access control and role management. Each task generating work items can assign a role to the work item, restricting this work item to participants who represent that role. Work items without a role become available to any participant connected to the system.
The external services of the WfMS consist of three parts. The first service is the workflow client interface, which manages the work item queue and distributes work items to connected clients. The second service is the process definition interface, which provides access to the model repository to allow the user to add and remove workflow models. Finally, the administration and monitoring service offers access to administrative and monitoring functions. The monitoring functions are especially critical for the testing system since they provide feedback regarding events happening during workflow execution.
This system provides a similar functionality to a workflow system in production use. In addition to this basic system, a workflow application client which simulates the behavior of workflow participants is needed to create an automated testing system which can execute tests of workflows without user intervention. This client is implemented as a BDI agent which connects to the three external service agents to the WfMS. The agent provides a user interface to the workflow engineer, which allows them to open the desired workflow model which is retrieved from the WfMS.
Since it is desirable for the workflow engineer to gain an overview of the parts of the workflow which are relevant to interaction with workflow participants, the client agent uses the workflow model to generate a tree representation of the workflow model which can be seen in Figure 2. The tree consists of a workflow node as the root node, which represents the workflow originally opened by the workflow engineer. If a workflow contains sub-workflows like BPMN plans, the children of its workflow node can include further workflow nodes representing those sub-workflows. The sub-workflow graph of a workflow can contain cycles, for example, when a sub-workflow uses its parent as a sub-workflow. This would suggest a graph representation to be the natural form for representing the workflow structure. However, cycles in the workflow graph are rare in practice and a workflow engineer would expect a tree form rather than a graph. Therefore the tree representation is more desireable, nevertheless the special case of a cyclic workflow structure should be supported. This problem is solved with the use of link nodes. If a particular sub-workflow is found again after having been found before, it is represented as a link node in the tree. This link node is a simple reference to the first occurance of the sub-workflow in the structure and no further expansion of the tree is done beyond this reference to avoid endless expansion.
If a workflow node is a BPMN workflow or workflow fragment its child nodes can, in addition to sub-workflow nodes, contain task nodes which represent tasks containing interaction with workflow participants. Lastly, the children of task nodes are parameter nodes, which represent typed parameters which would ordinarily be provided by a workflow participant.
The tree structure of a modelled workflow is presented to the workflow engineer in graphical form in the user interface. This allows them to define scenarios. Scenarios consist of sets of input values for each parameter of the workflow, definition of data collected from the workflow during the simulation and a validation function which is used to evaluate the success of the test and generate a report for the workflow engineer. Figure 3 demonstrates the use of scenarios in the full test procedure.
For each of the input parameters, the workflow engineer can add multiple values depending on the type of the parameter. The cartesian product of the input values in the scenario is used to create tests for the workflow which means that a minimum of a single value for each parameter is required for the scenario to generate at least a single viable test.
The workflow engineer can also select what is monitored during execution. This can include output values of the workflow, its final internal state, exceptions and the BPMN task elements executed. These values are passed to the validation function after execution to determine whether the test was a success and to generate a useful report. The validation function is also defined by the workflow engineer and included in the scenario. The validation function is defined in advance and is specified either by a Java class implementing the validation functionality or alternatively through the use of a number of configurable basic tests like comparison of result values with expected results.
Multiple scenarios can be defined for each workflow which can be automatically executed in succession. The total number of required test runs of the workflow are reported to the workflow engineer while assembling the scenarios. Since the cartesian product of multiple parameter values in a scenario quickly increases the complexity of the whole test, the workflow engineer has to carefully chose the tests and carefully balance between adding additional scenarios for the test run or adding additional parameter values to a single scenario.
The results of a test run can be reviewed in the report generated by the validation functions of the scenarios and is displayed to the workflow engineer in the client application. In addition, several tools provided by the Jadex platform can be used during the simulation as well. This includes and introspector tool for investigated the state of the workflow and a message center tool for monitoring messages between workflows and their support systems like the WfMS and the client application.
The next section will present an example use case for an industrial workflow used for change management as envisioned by Daimler AG and will demonstrate how the test system can be used to find implementation errors in advance of deployment.
Group Research and represents an industrial workflow used
for change management (see [
Since the workflow is very large and complex, this section will focus on a smaller subset of this workflow (cf. Figure 4). This subset involves the gathering of information about the planned change to the product, designating key personnel and assigning required resources. The final result of this part of the process is a description of the change request and requirements which will be used in the later part of the workflow to perform the change.
The process fragment contains a single goal for defining the change request. This goal is decomposed into subgoals, some of which contain context conditions which suspend their execution until the context has the required state for the goal to be adopted. Goals without further subgoals are associated with plans which are implmented by BPMN workflow fragments.
After implementation, one of the BPMN workflow fragments contained an error. The BPMN fragment containing the error was the fragment determining the parts affected by the change in the product. During the execution of the fragment, the leading developer responsible for the change request is required to enter the parts of the product which are affected by the change. The developer has the choice to do this using three different ways of providing this list of parts. The first way is to provide a list of serial numbers of the affected parts. The second way is to provide a drawing which is processed for part information and finally, the developer can give a structured description of the components affected by the change.
The first task lets the developer choose between those three ways of providing the part list (cf. Figure 5). The first task in this fragment generates a work item containing a list of three strings which represent the choices of the developer. The developer can select one of the strings and commits the work item. The string is then passed to the gateway, and compared to strings provided by the edges behind the gateway branch. If the string matches, the process continues executing using that path and provides the developer with a new work item which contains the information for the chosen method of entry.
The implementation error in this part of the process was that one of the strings provided by the edges did not match with the corresponding string generated by the entry type selection task. Since none of the edges on the gateway branch are marked as default edge which would be taken if no other edge matches, the workflow will terminate with an exception if the developer selects the faulty choice.
This error was found using the test system. A scenario had been created test each of the branches leaving the gateway in this workflow fragment. For every parameter in the workflow except the entry method choice of the developer a single value was added to the scenario. All three possible strings were then added to the parameter concerning the entry choice resulting in a scenario which specifically target this branch in the workflow (cf. Figure 6). In addition, example entries for the part specification had been added in order to verify the correct function of the workflow in identifying the parts affected by the change, test corner case entry such as empty strings for parts and test another branch in the workflow where the developer has to confirm the list of parts or otherwise cause a restart of the workflow fragment. The test complexity of the scenario required 972 test runs which were executed on an Intel i5 CPU clocked at 2.67GHz in less than a minute.
Some test runs resulted in an exception and the termination of the workflow. This event was logged during the simulation, including the parameter configuration used which caused the error. This result allowed the workflow engineer to find the fault in the workflow and correct the problem.
This paper has presented a simulation-based validation approach for workflows. The approach allows specifying execution scenarios in form of test cases, which include ranges of input values to be tested and defined output states to be reached for a successful execution. The approach is especially wellsuited for agile process descriptions with abstract specification means, such that possible process execution paths cannot easily be predicted. Nevertheless, the approach also works well with traditional process specification languages like BPMN. The implementation of the validation approach is based on the process execution facilities of the Jadex active component platform and provides additional tools for the specification, execution and validation of scenarios. The applicability of the approach has been exemplified by a case study from our project partner Daimler AG. It has been shown how the validation approach allows testing a modeled process to find and resolve existing problems in the process description.
Two interesting areas for future work are envisaged. First, the approach can be extended towards being used not only for process validation, but also for process analysis and optimization. Extending the simulation engine with elaborated analysis tools would allow measuring the quality of processes and benchmarking alternative processes against each other. Second, instead of using pre-specified test cases as scenarios, complex simulation models could be used to dynamically produce realistic test data. E.g. for logistics management processes, a simulation model of a supply chain could be connected to the process engine and provide input data for the workflow application to be tested.
Acknowledgement: We would like to thank the DFG for supporting the technology transfer project Go4Flex.