When assessing the usability of BPM technologies enterprises have to rely on vendor promises or qualitative data rather than on empirical or experimental research. To address this need Cheetah Experimental Platform (CEP) has been developed fostering experimental research on business process modeling. CEP provides components that are frequently used in controlled experiments and allows their assembly to experimental work ows. CEP supports experimental execution by mitigating risks endangering data validity through better user guidance. Additionally, CEP provides richer evaluation techniques compared to paper based experiments fostering the experiment's data analysis.
Providing e ective IT support for business processes has become an essential
activity of enterprises in order to stay competitive in today's market [
During our experimental research (e.g., [5{8]) we identi ed several typical problems in the di erent phases of experiments that might be addressed by appropriate tool support. In the experimental design phase the setup has to be de ned, including the de nition of objects, subjects and the execution order of di erent tasks. Providing components that are frequently used in controlled experiments (e.g., surveys, tutorials, process modeling tools) facilitate researchers the creation of experimental designs. Still, a successful experimental design largely depends on the experimenter's experience and knowledge of the domain. The second phase, experimental execution, highly bene ts from rich tool support as many risks endangering data validity can be mitigated through better user guidance (e.g., avoiding that subjects do not follow the experimental setup). Finally, tool support can also be bene cial in the experimental analysis phase as richer data evaluation techniques are available compared to paper based experiments (e.g., replaying the modeling process).
The remainder of this tool paper is structured as follows. Section 2 introduces a running example, which will be used in Section 3 for describing CEP. Finally, Section 4 concludes the paper with a summary and outlook on future work. 2
To illustrate the functionalities of CEP, we introduce a typical experimental
design as a running example (cf. Fig. 1). Let us assume that the goal of the
experiment is to investigate whether secondary notations (cf. [
Group 1 n/2 Participants
Group 2 n/2 Participants
Factor Level 1 Poor Layout Factor Level 2 Good Layout
Experimental Run
Change Task Process Model with Poor Layout Change Task Process Model with Good Layout
This section describes CEP. In particular, Section 3.1 illustrates how the platform can be used to support the design of experiments. Then, Section 3.2 deals with the actual operation of the experiment. Finally, Section 3.3 discusses how CEP fosters data analysis. 3.1
Even tough the creation of experimental designs is a task highly relying on researcher's experience and domain knowledge, tool support can be bene cial in
Investigating the Process of Process Modeling with Cheetah Experimental Platform 15 this phase. The majority of controlled experiments consists of a series of tasks that have to be executed by the experiment's subjects, referred to as Experimental Work ow. CEP enables experimenters to quickly assemble experimental work ows from components that have proven to work well in several experiments. In particular, CEP o ers a set of frequently used components, including surveys, tutorials and Cheetah Modeler for creating business processes (cf. Section 3.2).
The exemplary experimental work ow described in Section 2 is supported by CEP as illustrated in Fig. 2. Depending on the number of di erent groups several branches are available in the experimental work ow con guration. At the beginning of the experiment, subjects are provided with assignment sheets containing an introductory text, instructions for performing the modeling tasks and a group code. Irrespective of the code the subjects entered, each participant has to ll out a demographic survey before working through an interactive tutorial. Based on the group code the respective branch of the experimental work ow is entered, presenting subjects with a change task for a process model with good/bad layout. Finally, participants are asked to ll out a survey about the cognitive load of the performed change task. All activities of the experimental work ow are handled using components provided by CEP.
Enter Code
Demographic
Survey
Tutorial
Cognitive Load
Survey Group 1 Group 2
Modeling Task 1
Modeling Task 2 Experimental Work ow When executing the experimental work ow con guration CEP guides the user through the experiment ensuring that the setup is followed. Furthermore, data collected when executing the experimental work ow is stored on a central database server, giving researchers the possibility to check whether all activities were completed and to restore the experiment to a speci c state (e.g., in case of a crashed system). If the database server cannot be accessed a local copy is created and the user is asked to send it to the experiment's supervisor via email.
The experiment described in Section 2 is supported by CEP as follows. After entering the code identifying the group, the upcoming survey is collecting the user's demographic data. The survey ensures that all questions marked as mandatory are answered before the user continues with the next step in the experimental work ow. Before starting the actual modeling task the experimental work ow contains an interactive tutorial explaining the functionalities of Cheetah Modeler to make sure the used notation is well understood and participants know how to utilize the tool to change the process model. Therefore, each important functionality is presented by a screencast and users have to perform the corresponding modeling step. Depending on the entered code users are presented with process models with good/bad layout serving as a basis for the change task. Afterwards, a nal survey assessing the mental e ort for performing the change task is displayed.
Cheetah Modeler In order to enable the investigation of how process models are created, CEP o ers Cheetah Modeler, which is a rather simple modeling component providing only basic modeling functionalities for simulating a pen and paper modeling session using a subset of BPMN (cf. Fig. 3). The focus was put on developing a tool facilitating the investigation of how process models are created, rather than providing a full edged modeling suite. Currently, BPMN is the only process modeling language supported by CEP. Nevertheless, support for other notations was kept in mind when designing CEP and can easily be integrated. Logging: Besides monitoring the experiment's correct execution and gathering the results of surveys, the collection of data on how users create process models was one of the main objectives when implementing Cheetah Modeler.
Investigating the Process of Process Modeling with Cheetah Experimental Platform 17 Consequently, every change to the process model (e.g., add/delete/move activity, add/delete/move edge) and the corresponding timestamp is automatically recorded and stored in a separate process log, o ering the possibility for detailed investigations concerning the process of modeling (cf. Section 3.3). 3.3
In addition to e ciently executing and monitoring experiments, data analysis was one of the main objectives when developing CEP. This section sketches the provided functionalities of Cheetah Analyzer, o ering various data export features and means for replaying process models.
Experimental Work ow To be able to analyze data collected when
executing the experimental work ow an export system is in place. By providing the
option to export data as Comma-Separated Values (CSV) les, several tools for
performing statistical analysis can be addressed (e.g., SPSS, Excel).
Process of Process Modeling One of the main advantages of using CEP is the
possibility of replaying process models created with Cheetah Modeler. Recording
all modeling steps enables researches to investigate how business process models
are really created. For this purpose Cheetah Analyzer was implemented allowing
for a step by step execution of modeling processes (cf. Fig. 4). Additionally,
researches can export modeling processes using the Mining XML (MXML) format,
allowing them to apply process mining techniques using ProM [
In context of the experiment presented in Section 2 researchers can have a detailed look on how the given process models were changed and if the layout had an in uence on the change process. For example, it might be possible that users presented with a bad process layout rearranged activities before performing the actual change. 4
Cheetah Experimental Platform, described in this tool paper, supports researches in conducting controlled experiments on business process modeling. In particular, CEP provides a repository of typical components (e.g., surveys, tutorials, process modeling tools) which can be used for assembling experimental workows. Furthermore, the risk of producing invalid data is mitigated as the user is guided throughout the experiment's execution, reducing the number of accidental errors. In addition, richer analysis of data is possible compared to paper based experiments.
Future developments include a graphical experimental work ow and survey builder to further facilitate the creation of experimental designs as well as a dashboard simplifying the supervision of experiments. Furthermore, we would like to investigate the in uence of collaborative modeling on how process models are created. For this purpose, CEP is currently extended toward collaborative modeling support.
Acknowledgements: We thank Dirk Fahland, Jan Mendling, Hajo A. Reijers, Matthias Weidlich and Werner Wild for their much appreciated feedback when developing Cheetah Experimental Platform.