In many elds of science, the quality and applicability of presented approaches heavily relies on experimental results. At the same time, such experiments are only described brie y and are typically boiled down to a single diagram or table. To maintain high scienti c standards, experiments need to be transparent and repeatable. This not only provides a deeper understanding in the used tools, but also facilitates the improvement and comparison of approaches. This paper introduces service-technology.org/live, a Web site which allows to replay experiments in a Web browser. This gives colleagues and reviewers the possibility to get detailed information on the experimental setup without the need of actually installing tools. The Web site further o ers the used tools and the input data for download together with information on how to repeat the experiments locally.
The formal veri cation of business process models forms a constant stream of
research in the BPM community. With a variety of correctness criteria of which
soundness is the most popular one, also a multitude of automatic veri cation
tools were developed over the last years. The algorithms implemented by these
tools typically have a devastating worst-case complexity (usually an exponential
explosion in the size of the model) which seems to make any application beyond
toy examples unreasonable. As a matter of fact, experiences in the eld of model
checking [
To follow good scienti c standards, such experiments should not only be thoroughly described. To achieve maximal transparency and to simplify comparison and improvements, they also need to be repeatable. That said, not only the respective tools and input data need to be published, but also information on the installation and invocation are required to replay experiments. Unfortunately, only few papers provide such details and experimental results are typically boiled down to a few tables or diagrams, making the entire experiment and hence the implemented algorithm intransparent and vulnerable.
We identi ed several problems that hinder adequate transparency: { Page limits. Technical details on the installation or invocation of tools can hardly be provided within the page limit of a typical research paper. At the same time, prototypes are often a byproduct of the respective research paper and hence dedicated tool papers are not yet available. { Availability of tools. Many tools are proof-of-concept implementations for the respective research paper and are often not publicly available. The same holds for beta versions or commercial tools. { Installation of tools. Scienti c software development often lacks a professional release and documentation process which makes the installation complex and cumbersome. { Description of the experimental setup. An experiment hardly involves a single call of a tool or \click" of a button. Often, only the interplay between several tools or parameters realize the experiment. { Availability of input data. Even if all tools are available and the experimental setup is properly described, the concrete input data are hardly available for download. This is typically because of nondisclosure agreements or other legal constraints on industrial process models. But even if input data are public (e.g., published reference models), they are frequently converted beforehand or otherwise preprocessed.
This paper introduces service-technology.org/live, a Web site dedicated to the technical aspects of the described problems. This Web site not only \outsources" technical descriptions on the experimental setup, but also facilitates the replay of the experiments within a Web browser. The remainder of the paper is organized as follows. The next section gives an overview of service-technology.org/live. Section 3 discusses technical details. Finally, Sect. 4 gives directions of future work and concludes the paper. 2
The Web site service-technology.org/live hosts with BPEL2oWFN [
In the simplest case, the user can choose an experiment which is described in the respective paper (i.e., choosing the name of a model from a drop-down list) and let the experiment be run \live" on the Web server. For other experiments, also a few parameters can be set. The output (see Fig. 2 for an example) consist of the following elements: { a description of the input of the experiment (e.g., the origin of the model), { console windows (i.e., the output of the command-line tools), { generated graphical representations (e.g., Petri nets or automata), { a summary of invoked tools with their parameters, and { download information for the involved models and tools.
Admittedly, the current state of interactivity is limited. We believe, however, that even this simplistic Web site already provides a decent amount of transparency for the experimental results. With a hosted experiment, we give colleagues and reviewers not only the opportunity to check for the existence of the tools, but also give an idea how the results are generated and also show the \look and feel" of the tools without the need of downloading or installing. This is, however, still possible, because all tools and input models are available online.
The Web site service-technology.org/live is written in PHP, but any other language that allows the Web server to invoke locally installed tools could be used instead.
The current server is an Apple Mac mini (1.83 GHz, 1 GB RAM) running an Apache 2.2 server with PHP 5.3.
The main components such as console windows or graphics generation and presentation have been encapsulated to facilitate extensions and the integration of new tools and experiments. The free JavaScript library jQuery1 further realizes basic UI concepts such as tabbed windows and asynchronous loading of images using AJAX. The code of the Web site is free and open source.2
New experiments can be quickly integrated by performing two tasks: 1. Add a form to the front page to choose the input model and parameters. 2. Create a PHP script that runs and presents the experimental results.
Both steps are highly generic, and because of the encapsulation of server and language-speci c details, we claim that anyone who is able to execute the tools in a console is likewise able to create such les for new experiments | assuming basic knowledge on HTML and PHP.
In the current state, no user access controls are implemented. All experiments are open for everybody. It is, however, possible to upload input models to the 1 http://jquery.com/ 2 http://svn.gna.org/svn/service-tech/trunk/meta/live/ server or to provide a URL to a model. Such uploaded models are processed just like the hosted models, but are not shared with other users, and are deleted from the server regularly. 4
Concrete experiments hosted by service-technology.org/live have already been
referenced in several published research papers [
In future work, we plan to continue to simplify the addition of new experiments
and tools. We also work on decoupling the hosted tools from the Web site and to
o er them as software as a service using Web service standards such as SOAP
and WSDL. Thereby, we can wrap complex veri cation tasks and allow them
to be invoked remotely over the Web from other tools. We currently evaluate
the use of the Petri net model checker LoLA [