Motivation: In many media applications, there is a need to index very large amounts of audiovisual content in a minimum of time. Due to their scalability, Grid infrastructures are well suited for processing a large number of media files. Within the BMBF1-funded research project MediaGrid 2, a distributed media analysis platform is currently being developed. It enables its users to analyze and index large quantities of multimedia data within a short amount of time. Grid services as distributed processing units and BPEL4Grid [10] as a scientific BPEL3-based workflow system are the foundations of the platform. To reduce the interface complexity and make the platform user-friendly, additional components have been added. This paper describes the architecture of the media analysis platform and the components involved. Results: An architecture offering a generic job interface in combination with an extensible BPEL repository is presented. The complexity of the system is hidden from the user and shifted to the platform provider. Thereby, the barrier to use the system as an end user is lowered to a minimum. Availability: Parts of the components that are combined to form the media analysis platform already exist (for instance, the generic job interface and the BPEL engine). Further enhancements such as the BPEL Repository, the BPEL Adapter Repository and the Mapping and Invocation Component will be integrated into the platform and will be made available in 2011. Contact: benjamin.ihle@iais.fraunhofer.de
The MediaGrid research project deals with distributed media analysis in Grid environments. The aim of the project is to build a Grid infrastructure that provides distributed services (encapsulating adequate algorithms) for analyzing audio and video content.
In MediaGrid, media analysis is conducted based on several
processing steps (implemented as distributed Grid services) that
are orchestrated using workflow technology. The services can be
arranged differently in terms of their order of processing. To
manage various sequences of distributed Grid services, MediaGrid
1 Bundesministerium fu¨r Bildung und Forschung (German Federal Ministry
of Education and Research)
2 http://www.mediagrid-community.de
3 Business Process Execution Language
uses the Business Process Execution Language (BPEL) – the de
facto standard for business process modeling in today’s enterprises.
Moreover, BPEL is already widely used for the implementation and
execution of workflows in scientific applications [
This paper presents a system architecture for performing media analysis via a BPEL-based service platform in Grid environments. As a key feature, it shifts the complexity of a workflow system for distributed job processing from the customer to the platform provider. At the same time, the entire functionality provided by the Grid service remains available for the customer to the full extent. This is achieved by offering a generic interface that simplifies the usage of distributed services and significantly lowers the barrier to entry for the user to integrate the provided services in his or her own workflow. The paper gives an overview of the software components that are part of the workflow management system for service-oriented and distributed media applications.
This paper is organized as follows. Section 2 provides a general overview of the involved components and briefly describes their interaction. In Section 3, the BPEL4Grid engine used to execute Grid-enabled workflows is presented. Section 4 describes the components responsible for workflow management, while Section 5 depicts the user interface for job submission and monitoring. Projects related to the work presented in this paper are described in Section 6. Finally, Section 7 concludes the paper and outlines areas for future research. 2
In this section, the architecture of the system is described from
a bird’s-eye view to illustrate the interplay of the different
components. The architecture includes several components, as
depicted in Figure 1. The system’s foundation is a service-oriented
architecture based on Web and Grid services orchestrated using
BPEL workflows. Since BPEL has a number of drawbacks with
respect to WSRF-based Grid services, the Grid-enabled workflow
modeling tool Domain-Adaptable, Visual BPEL4WS Orchestrator
(DAVO) [
The BPEL4Grid Engine only provides a workflow’s interface description (WSDL document), but no additional metadata like information on parameters. This data is required to fulfill the main goal of this work: ease the invocation of the deployed workflows. Therefore, we introduce an additional component for Copyright c 2011 for the individual papers by the papers’ authors. Copying permitted only for private and academic purposes.
Developer
Job Type → BPEL URI workflow registration: the BPEL Repository. Besides providing documentation on available workflows, it is also responsible for creating and storing Web Service stubs used to invoke the deployed workflows.
To invoke a workflow using a stub, an adapter object has to be provided that is capable of mapping the parameters submitted by the user to the generated stub. The BPEL Adapter Repository holds such an adapter for every available workflow.
The actual mapping and workflow invocation is performed by the BPEL Mapping and Invocation Component (BMIC). Every request to start the processing of a workflow is internally managed as a separate job object that contains information on the workflow to be invoked and its parameters. These job objects are organized in a Job Queue. When a new job is selected for processing from the Job Queue, the BMIC assembles all data objects necessary to invoke the workflow. Finally, it triggers the processing of the workflow.
Despite the variety of available workflows, there is only a single Web Service interface through which workflows can be started, managed and monitored: the Job Service. The Job Service interface defines a data type for submitting a Generic Job Description. It contains all the information needed to invoke a BPEL workflow such as the Job Type and Parameters. Users can query the Job Service for available workflows, choose a workflow they want to utilize and compose a Generic Job Description before it is submitted.
Furthermore, the Job Service provides operations for monitoring a job’s execution status and manage the execution lifecycle, i.e. to pause, resume or abort a running job.
BPEL4WS has emerged from the earlier proposed XLANG4 and Web Service Flow Language (WSFL)5. It enables the construction of complex web services composed of other web services that act as the basic activities in the process model of the newly constructed service. Access to the process is exposed by a execution engine through a web service interface, allowing those processes to be accessed by web service clients or to act as basic activities in other processes. BPEL features several basic activities to interact with the services being arranged in the workflow. These activities cover service interaction, data manipulation and the manipulation of the control flow. Structured activities, ranging from a sequential ordering of activities up to building directed acyclic graphs, allow the composition of complex operations. Furthermore, BPEL includes the features of scoping activities and specifying fault handlers and compensation handlers for scopes.
To communicate with stateless and stateful services that are built
on top of the Web Services Resource Framework (WSRF)6, an
extended version of a BPEL engine [
The GCRI activity identifies the corresponding partner link set before determining the resource link from it and constructing an invoke object. The created invoke object is added to the execution queue of the engine. As soon as the invoke is dequeued, any resource link information contained in that invoke is identified and the concrete endpoint is set in the Axis call. As soon as the response arrives, it is parsed and the resource key as well as the endpoint address are stored as a resource in the resource link. This information is handled by the partner link set data structure. When GI is called, it performs a lookup for the partner link set and identifies the resource link corresponding to it. The engine’s natural strategy to resolve a partner link is to look them up by their names. To use this mechanism, we decided to introduce unique identifiers for resources that are stored in the invoke object. Hence, the engine can resolve the resource link during the creation of an Axis call object. Subsequently, the correct endpoint information is saved within the call and the information about the resource key is put into the message context. GDRI is used when a WSRF resource is not required anymore and therefore its lifetime should end. It constructs an invoke object in the same way GI does (but with the intention of destroying the resource).
Another obstacle BPEL workflows have to face are security
constraints: BPEL has a number of drawbacks with respect to
the more complex world of WSRF-based Grid computing. The
BPEL security concept is not equipped to deal with complex
multi-protocol Grid environments and does not integrate with the
Grid Security Infrastructure (GSI). While BPEL is mainly focused
on anonymous HTTPS-based TLS security or manual role-based
authentication encoded in SOAP headers, Grid computing has a
mandatory user-centric security approach using X.509 certificates
that far exceeds the scope and capability of the BPEL security
model. The Grid extensions mentioned above also feature the
integration of GSI into the Grid service invocation that can also
manage the lifetime of proxy certificates [
Due to the fact that BPEL is an XML-based web service composition language, the definition of a workflow is quite laborious and time-consuming without tool support. As a consequence, several research and commercial activities concentrate on the design and development of graphical BPEL workflow editors focusing on a clear visualization and syntactically correct mapping of the graphical representation to BPEL4WS code. Thus, existing BPEL workflow editors are only suitable for web service experts who are familiar with the details of web services and BPEL4WS. Non web service experts are normally overburdened by these editors.
The Domain-Adaptable, Visual BPEL4WS Orchestrator (DAVO)
[
The architecture of DAVO is based on a model-driven approach
and follows the model-view-controller (MVC) design pattern [
The mapping from the internal data model to executable BPEL code is performed by a code generation component. It generates at least three files from the internal model: a bpel file that contains the logic of the workflow, a WSDL file with the workflows interface description and a (non-standardized) deployment descriptor that contains runtime information such as service endpoints. 4 4.1
When BPEL processes are deployed to a BPEL engine, interfaces are needed to handle workflow discovery and execution. The BPEL4Grid engine, which in turn is based on ActiveBPEL8, is used in the MediaGrid project; it provides such a simple web service interface that can be used to discover available workflows, monitor their execution and manage the lifecycle of running processes. However, this simple interface lacks the possibility to obtain additional information to a process such as process semantics or a documentation of the process parameters. Furthermore, there is no single interface through which workflows that are hosted in remote BPEL engines can be discovered. What is needed is a centralized workflow repository capable of providing information about locally and remotely deployed processes to manage and discover available workflows at a single location.
Due to these requirements, a separate workflow registry component is used: the BPEL Repository (BR). Acting as a registry for BPEL processes, the BR stores information about all available processes. Similar to the management interface of the BPEL4Grid engine, processes can be added, altered or removed via its own management web service interface. It can also be used for process deployment, such that additional documentation about the process and its parameters can be provided along with the deployment.
Besides the workflows that run in the local BPEL engine, external workflow providers can use the interface to offer remotely hosted BPEL workflows to the user. All they need to provide is a WSDL interface description of the process. Additionally, they can add
documentation about the process and its parameters that can be presented to the user when he or she intends to use the workflow.
Based on the WSDL interface description of a BPEL process, the BR is able to automatically generate web service stubs for the corresponding interface. These stubs are administered by the BR and are used by the BPEL Mapping and Invocation Component described in Subsection 4.3 to invoke the BPEL process. 4.2
The BPEL Adapter Repository (BAR) stores the necessary objects for mapping the Job Type of a Generic Job Description to a specific BPEL process. For every Job Type, there exists a corresponding BPEL process that can be invoked by its BPEL WS9 and that is accessible via the BPEL WS URL. As shown in Figure 1, the BAR returns an URI that specifies the location of the corresponding BPEL WS URL within the BR. To perform the mapping between the parameters specified by the Generic Job Description and the input variables of a certain BPEL WS, the repository additionally contains a BPEL Adapter (BA) for each BPEL WS, which contains all required mapping information. Thus, for every Job Type specified by the customer, the BAR provides an URI and a BA. Thereby, the repository only stores the mapping information but does not do any mapping at all. The mapping is actually done by the BPEL Mapping and Invocation Component that is described in Section 4.3.
The information about how to map a Generic Job Description to a specific BPEL WS including its input parameters has to be created manually. Since the mapping information can neither be extracted automatically from a Generic Job Description nor from a BPEL WS, it has to be implemented by a developer who knows both workflow descriptions and is aware of how the parameters have to be assigned to each other. Figure 1 shows the role of the developer. He or she is responsible for implementing the mapping of Job Types to their corresponding BPEL URIs and BAs. To lower the barrier to entry for the customer, this relatively expensive and time consuming task is concealed from him or her. Thus, the effort to integrate the functionality of the platform in the customer’s environment is reduced to a minimum. 4.3
The BPEL Mapping and Invocation Component (BMIC) performs the mapping between the data of the Generic Job Description provided by the customer and the associated BPEL WS. Moreover, it invokes the corresponding process. To know how to map the parameters to the input variables of the specific BPEL WS, a retrieval of various data objects is necessary beforehand. Thereby, the user’s input data in the form of a Generic Job Description and the WSDL URL of the corresponding BPEL WS as well as an appropriate BA are used. These objects are retrieved from internal repositories. The entire process of data mapping is described in more detail below. Figure 2 presents a graphical overview of the mapping process showing its single steps in numerical order.
A new job processing is triggered by the customer by providing a new Generic Job Description. It comprises a Job Type that corresponds to a certain workflow type and a number of parameters describing necessary input values. First (1), the BMIC needs to
know which specific BPEL process belongs to the Job Type provided by the customer. Therefore, it puts a request to the BAR specifying the Job Type that the user has chosen. The BAR matches the provided Job Type to an URI pointing to the location of the BPEL WS to call (2) and returns this URI along with the corresponding BA (3). To assemble a web service client to invoke functions of the destined BPEL WS, a request is put to the BR providing the URI of the mapped BPEL workflow (4). The BR does the matching for the provided URI (5) and returns the BPEL WS URL that points to the assigned BPEL WS (6). Apart from the BPEL WS URL, the BR provides WS Stubs generated from the WSDL description of the BPEL WS URL. Now that all necessary information is available to create an user-specific request to the corresponding BPEL WS, the actual mapping process takes place (7). The parameters originally provided by the customer through the Generic Job Description are mapped to the input variables of the WS Stubs of the assigned BPEL WS. The result of the mapping process is a parametrized web service client. It is used to invoke the BPEL WS that has been identified for the user-chosen Job Type. Finally, the corresponding BPEL WS is invoked (8) and the processing of the workflow steps begins. 5
BPEL processes mostly represent a chain of sequential and parallel processing steps and are invoked using web service interfaces. Each process provides such an interface specifying the operations available and the parameters needed to invoke the operation10. Since BPEL processes can differ significantly depending on the task they perform, the corresponding web service interfaces can also differ. A user would have to implement a specific client for every new process he or she may want to use. A drawback of this approach is that a growing number of different processes increases the complexity and the effort to integrate them into an application.
The main goal of the Job Service interface is to bypass this shortcoming of a service-oriented architecture by providing a single, generic web service interface for workflow invocation. This interface acts as a fac¸ade to hide the workflow invocation logic from the user by transferring the complexity towards the platform provider. Instead of choosing a workflow and generating stubs to invoke this workflow, the user composes a Generic Job Description which is interpreted by the Job Service as shown in Figure 1. The job description contains a specific Job Type and the Parameters necessary for its execution. An extraction of the XML schema code representing the job description is shown in Listing 1.
Every Job Type uniquely maps to a corresponding BPEL WS able to perform the desired task. By using the mapping component described in Subsection 4.3, the parameters provided are matched to the corresponding BPEL parameters. Since stub classes are created automatically by the BPEL Repository, applications using the platform only have to implement a single web service interface.
The Job Service interface cannot only be used for job submission. It also provides all the information necessary to compose a job description. Applications can query the Job Service for available Job Types and present them to the user. The customer can easily choose a Job Type from a number of already existing Job Types 10 For simplification, we assume that a workflow has only one operation. “Invoking a workflow” means invoking this single operation. 1 Job Type
3
BPEL URI BPEL Adapter 2 Job Type
↓ BPEL URI
BPEL Adapter
Repository and has to provide required parameters. After a job description has been composed, it is passed to the Job Service that in turn starts the workflow processing. The Job Service responds with a unique job ID which can be used to manage and monitor the job execution process. Besides the ability to add new jobs, the Job Service interface provides operations to query a job’s state and to abort, suspend or resume a running job using the job ID. <complexType name=” j o b D e s c r i p t i o n ”> <s e q u e n c e> <e l e m e n t name=” j o b T y p e ” t y p e =” t n s : j o b T y p e ” /> <e l e m e n t name=” t i t l e ” t y p e =” s t r i n g ” /> </ s e q u e n c e> </ complexType> <complexType name=” j o b T y p e ”> <s e q u e n c e> <e l e m e n t name=” comment ” t y p e =” s t r i n g ” /> <e l e m e n t name=” p a r a m e t e r s ” t y p e =” t n s : j o b P a r a m e t e r ” minOccurs =” 0 ” maxOccurs=” unbounded ” /> <e l e m e n t name=” t y p e ” t y p e =” s t r i n g ” /> </ s e q u e n c e> </ complexType> <complexType name=” j o b P a r a m e t e r ”> <s e q u e n c e> <e l e m e n t name=” comment ” t y p e =” s t r i n g ” /> <e l e m e n t name=” name ” t y p e =” s t r i n g ” /> <e l e m e n t name=” t y p e ” t y p e =”QName” /> <e l e m e n t name=” v a l u e ” t y p e =” b a s e 6 4 B i n a r y ” /> </ s e q u e n c e> </ complexType> Listing 1. XML schema code representing the Job Description
After the Job Service has received a job description, a new job is created and added to the Job Queue from where it is further processed. Hence, the Job Service decouples the asynchronous submission of jobs from the synchronous job execution. Jobs can be submitted independently of job execution. With the job ID, users can determine a job’s state and manage its execution lifecycle. Since the BPEL Engine is in control of process execution, operations like 6 BPEL WS
URL WS Stubs
4 BPEL URI
BPEL Repository
5 BPEL URI
↓ BPEL WS URL
WS Stubs suspend, resume or abort are forwarded to the engine’s management interface.
More complex information, e.g. about the implementation of certain BPEL processes, remains hidden from the user. Thus, the effort to integrate the functionality of the platform in a user’s own environment is reduced to a minimum. 6
The CASSANDRA framework [
Blower et al. [
With a large scale content analysis engine, Gibbon et al. [
An architecture and an API to support a distributed data flow
model in conjunction with a centralized control flow is presented
by Barker et al. [
In this paper, we have presented an architecture for a generic Grid workflow management platform applied to media analysis. By specifying a single Job Service interface, the barrier to entry for using the workflows is lowered to a minimum without losing any functionality. No matter how many different workflows the customer likes to process, just one interface has to be implemented. The complexity of workflow processing is transferred from the customer to the platform provider. Acting as a fac¸ade for workflow processing, the Job Service aggregates all functions needed to invoke, manage and monitor workflows. Furthermore, external workflows can be added to the BPEL Repository to support the integration of workflows operated by third party providers.
The presented solution is currently under development and is estimated to be available as a prototype in 2011. Areas for future work are: a) completing the implementation, b) using the architecture for various analysis applications, and c) evaluating the performance and scalability of the proposed approach.