=Paper=
{{Paper
|id=None
|storemode=property
|title=service-technology.org/live - Replaying tool experiments in a Web browser
|pdfUrl=https://ceur-ws.org/Vol-615/paper23.pdf
|volume=Vol-615
|dblpUrl=https://dblp.org/rec/conf/bpm/Lohmann10
}}
==service-technology.org/live - Replaying tool experiments in a Web browser==
https://ceur-ws.org/Vol-615/paper23.pdf
service-technology.org/live
Replaying tool experiments in a Web browser
Niels Lohmann
Universität Rostock, Institut für Informatik, Rostock, Germany
niels.lohmann@uni-rostock.de
Abstract. In many fields of science, the quality and applicability of
presented approaches heavily relies on experimental results. At the same
time, such experiments are only described briefly and are typically boiled
down to a single diagram or table. To maintain high scientific standards,
experiments need to be transparent and repeatable. This not only pro-
vides a deeper understanding in the used tools, but also facilitates the
improvement and comparison of approaches.
This paper introduces service-technology.org/live, a Web site which allows
to replay experiments in a Web browser. This gives colleagues and review-
ers the possibility to get detailed information on the experimental setup
without the need of actually installing tools. The Web site further offers
the used tools and the input data for download together with information
on how to repeat the experiments locally.
1 Motivation
The formal verification of business process models forms a constant stream of
research in the BPM community. With a variety of correctness criteria of which
soundness is the most popular one, also a multitude of automatic verification
tools were developed over the last years. The algorithms implemented by these
tools typically have a devastating worst-case complexity (usually an exponential
explosion in the size of the model) which seems to make any application beyond
toy examples unreasonable. As a matter of fact, experiences in the field of model
checking [1] reveal that even theoretically unfeasible algorithms may perform sur-
prisingly well on real-world instances. Consequently, prototype implementations
need to be validated with the help of case studies or experiments.
To follow good scientific standards, such experiments should not only be
thoroughly described. To achieve maximal transparency and to simplify compari-
son and improvements, they also need to be repeatable. That said, not only the
respective tools and input data need to be published, but also information on the
installation and invocation are required to replay experiments. Unfortunately,
only few papers provide such details and experimental results are typically boiled
down to a few tables or diagrams, making the entire experiment and hence the
implemented algorithm intransparent and vulnerable.
We identified several problems that hinder adequate transparency:
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
– Page limits. Technical details on the installation or invocation of tools can
hardly be provided within the page limit of a typical research paper. At the
same time, prototypes are often a byproduct of the respective research paper
and hence dedicated tool papers are not yet available.
– Availability of tools. Many tools are proof-of-concept implementations for
the respective research paper and are often not publicly available. The same
holds for beta versions or commercial tools.
– Installation of tools. Scientific software development often lacks a professional
release and documentation process which makes the installation complex and
cumbersome.
– Description of the experimental setup. An experiment hardly involves a single
call of a tool or “click” of a button. Often, only the interplay between several
tools or parameters realize the experiment.
– Availability of input data. Even if all tools are available and the experimental
setup is properly described, the concrete input data are hardly available for
download. This is typically because of nondisclosure agreements or other legal
constraints on industrial process models. But even if input data are public
(e.g., published reference models), they are frequently converted beforehand
or otherwise preprocessed.
This paper introduces service-technology.org/live, a Web site dedicated to the
technical aspects of the described problems. This Web site not only “outsources”
technical descriptions on the experimental setup, but also facilitates the replay of
the experiments within a Web browser. The remainder of the paper is organized
as follows. The next section gives an overview of service-technology.org/live.
Section 3 discusses technical details. Finally, Sect. 4 gives directions of future
work and concludes the paper.
2 Overview
The Web site service-technology.org/live hosts with BPEL2oWFN [6], Fiona [10],
Wendy [7], Rachel [5], Marlene [3], and Mia [4] several of the tools developed
by the theory of programming research groups at the University of Rostock and
the Humboldt-Universität zu Berlin. Their focus is the correctness of business
process and service models. We provide a detailed overview in [9]. As the tools
are command-line based, their usage is — compared to tools with a graphical user
interface — rather complicated. Furthermore, the tools follow the “one purpose,
one tool” philosophy and can be combined in many ways to perform higher level
tasks. To still be able to reproduce experimental results, we fixed several tool
setups and offer them at service-technology.org/live, see Fig. 1.
In the simplest case, the user can choose an experiment which is described in
the respective paper (i.e., choosing the name of a model from a drop-down list)
and let the experiment be run “live” on the Web server. For other experiments,
also a few parameters can be set. The output (see Fig. 2 for an example) consist
of the following elements:
Fig. 1. Start page of service-technology.org/live.
– a description of the input of the experiment (e.g., the origin of the model),
– console windows (i.e., the output of the command-line tools),
– generated graphical representations (e.g., Petri nets or automata),
– a summary of invoked tools with their parameters, and
– download information for the involved models and tools.
Admittedly, the current state of interactivity is limited. We believe, how-
ever, that even this simplistic Web site already provides a decent amount of
transparency for the experimental results. With a hosted experiment, we give
colleagues and reviewers not only the opportunity to check for the existence of
the tools, but also give an idea how the results are generated and also show the
“look and feel” of the tools without the need of downloading or installing. This is,
however, still possible, because all tools and input models are available online.
3 Technical realization
The Web site service-technology.org/live is written in PHP, but any other language
that allows the Web server to invoke locally installed tools could be used instead.
Fig. 2. Example for a console output.
The current server is an Apple Mac mini (1.83 GHz, 1 GB RAM) running an
Apache 2.2 server with PHP 5.3.
The main components such as console windows or graphics generation and
presentation have been encapsulated to facilitate extensions and the integration
of new tools and experiments. The free JavaScript library jQuery1 further realizes
basic UI concepts such as tabbed windows and asynchronous loading of images
using AJAX. The code of the Web site is free and open source.2
New experiments can be quickly integrated by performing two tasks:
1. Add a form to the front page to choose the input model and parameters.
2. Create a PHP script that runs and presents the experimental results.
Both steps are highly generic, and because of the encapsulation of server and
language-specific details, we claim that anyone who is able to execute the tools
in a console is likewise able to create such files for new experiments — assuming
basic knowledge on HTML and PHP.
In the current state, no user access controls are implemented. All experiments
are open for everybody. It is, however, possible to upload input models to the
1
http://jquery.com/
2
http://svn.gna.org/svn/service-tech/trunk/meta/live/
server or to provide a URL to a model. Such uploaded models are processed just
like the hosted models, but are not shared with other users, and are deleted from
the server regularly.
4 Conclusion
Concrete experiments hosted by service-technology.org/live have already been
referenced in several published research papers [5,8,7]. We feel committed to use
service-technology.org/live as a part of any future research paper which involves
experimental results. By encapsulating repeatedly used functionality such as
console windows or graphics generation, new results can be added without much
programming effort. By publishing the source code of service-technology.org/live
as open source, we hope that other research groups follow our example.
In future work, we plan to continue to simplify the addition of new experiments
and tools. We also work on decoupling the hosted tools from the Web site and to
offer them as software as a service using Web service standards such as SOAP
and WSDL. Thereby, we can wrap complex verification tasks and allow them
to be invoked remotely over the Web from other tools. We currently evaluate
the use of the Petri net model checker LoLA [11] as soundness checker for the
modeling platform Oryx [2].
References
1. Clarke, E.M., Grumberg, O., Peled, D.A.: Model Checking. MIT Press (1999)
2. Decker, G., Overdick, H., Weske, M.: Oryx - an open modeling platform for the
BPM community. In: BPM 2008. pp. 382–385. LNCS 5240, Springer (2008)
3. Gierds, C., Mooij, A.J., Wolf, K.: Reducing adapter synthesis to controller synthesis.
IEEE T. Services Computing (2010), (accepted for publication in July 2010)
4. Liske, N., Lohmann, N., Stahl, C., Wolf, K.: Another approach to service instance
migration. In: ICSOC 2009. pp. 607–621. LNCS 5900, Springer (2009)
5. Lohmann, N.: Correcting deadlocking service choreographies using a simulation-
based graph edit distance. In: BPM 2008. pp. 132–147. LNCS 5240, Springer
(2008)
6. Lohmann, N.: A feature-complete Petri net semantics for WS-BPEL 2.0. In: WS-FM
2007. pp. 77–91. LNCS 4937, Springer (2008)
7. Lohmann, N., Weinberg, D.: Wendy: A tool to synthesize partners for services. In:
PETRI NETS 2010. pp. 297–307. LNCS 6128, Springer (2010)
8. Lohmann, N., Wolf, K.: Petrifying operating guidelines for services. In: ACSD 2009.
pp. 80–88. IEEE Computer Society (2009)
9. Lohmann, N., Wolf, K.: How to implement a theory of correctness in the area of
business processes and services. In: BPM 2010. LNCS, Springer (2010), (in press)
10. Massuthe, P., Weinberg, D.: Fiona: A tool to analyze interacting open nets. In:
AWPN 2008. pp. 99–104. CEUR Workshop Proceedings Vol. 380, CEUR-WS.org
(2008)
11. Wolf, K.: Generating Petri net state spaces. In: ICATPN 2007. pp. 29–42. LNCS
4546, Springer (2007)