-VisIVO Science Gateway is a web based, workflow enabled environment wrapped around a WS-PGRADE/gUSE portal integrating seamlessly large-scale multi-dimensional astrophysical datasets with applications for processing and visualization based on Distributed Computing Infrastructures (DCIs). We present the main tools and services supported including an application for mobile access to the gateway. We discuss issues in sharing workflows and report our experiences in supporting specialised communities. We present a number of workflows developed recently for visualization and numerical simulations and outline future workflows currently under development. Finally, we summarise our work on the gateway with pointers to future developments.
I. INTRODUCTION
Visualization can play an important role in the context
of large-scale multi-dimensional astrophysical datasets, e.g.
in understanding, interpreting and verifying their intrinsic
characteristics [
Science gateways provide services to support searching,
managing and uploading/downloading (thus allowing sharing)
of applications and datasets. They enable user communities
to deploy their applications through common graphical user
interfaces, thus allowing scientists to focus on the actual
applications instead of learning and managing the required
infrastructures. The processes supported by gateways are organized
as scientific workflows [
This paper reports on the on-going developments of
VisIVO Science Gateway1 and VisIVO Mobile application, first
presented in [
A number of customized workflows is configured by default to allow local or remote uploading of datasets, datasets filtering and creation of scientific movies. These workflows are provided with specific user interface portlets to enable intuitive parameter setting for standard users while hiding the complexity of the underlying system and infrastructures. The mobile application employs user accounts from the gateway and offers a handy platform for astrophysical communities to share results and experiences of analysis and exploration of their datasets.
For displaying 2D or 3D plots, astrophysicists typically
deploy software packages programs such as Gnuplot,
SuperMongo, or scripting languages such as Python, Matlab or
IDL. VisIt3 or ParaView4 offer a combination of 2D and 3D
plotting capabilities, real-time and offline analysis, scripting
and graphical control. VisIt has been provided with grid
services for scientific collaborative visualization in UNICORE
Grids [
Nevertheless scientific visualization can be a fairly complex process involving several steps, e.g. filtering data, choosing a representation and desired level of interactivity and customizing the manner in which the data is displayed. None of the aforementioned tools are provided with a science gateway to interface them with workflow services. Within VisIVO Science Gateway and VisIVO Mobile ready to-use workflows can be downloaded, parametrized and executed under a controlled 2http://www.guse.hu 3https://wci.llnl.gov/codes/visit 4http://www.paraview.org environment. The visualization and filtering parameters can be chosen interactively and the workflow configuration and submission to DCIs is performed without exposing technical details so that end users can focus on their applications instead of devoting efforts in learning and managing the underlying infrastructures.
II.
VISUALIZATION TOOLS
VisIVO [
VisIVO Desktop [
Users of each realization can obtain meaningful visualizations rapidly while preserving full and intuitive control of relevant visualization parameters. This section focuses on VisIVO Server6 which can be installed on any web server with a database repository and contains the following distinct modules: VisIVO Importer, VisIVO Filters and VisIVO Viewer (see Figure 1).
VisIVO Importer converts user-supplied datasets into VisIVO Binary Tables (VBTs). A VBT is a highly-efficient data representation realized through a header file containing all necessary metadata and a raw data file storing actual data values. VisIVO Importer supports conversion from several popular formats such as: ASCII and CSV, VOTables or FITS Tables without imposing any limits on sizes or dimensionality. VisIVO Filters is a collection of data processing modules to modify a VBT or to create a new VBT from existing VBTs. The filters support a range of operations such as scalar distribution, mathematical operations or selections of regions. VisIVO Viewer is the visualization core component based on the Visualization ToolKit7. It creates 3D images from multi-dimensional datasets rendering points, volumes and isosurfaces. Moreover there is support for customized look up tables and visualizations using a variety of glyphs, such as cubes, spheres or cones. VisIVO Viewer can be also used to produce images in a given sequence of azimuth, elevation, and zooming values that can be externally mounted to produce movies.
To create customized renderings from astrophysical data tables VisIVO Importer is first utilized to convert user datasets into VBTs. Then, one or more VisIVO Filters can be applied to process these datasets, and finally VisIVO Viewer is invoked to display these renderings. Figure 1 illustrates the typical sequence of steps required within the VisIVO Server processing pipeline.
5http://glite.cern.ch 6http://sourceforge.net/projects/visivoserver 7http://www.vtk.org
The existing VisIVO Web [
The VisIVO Science Gateway is designed as a workflow enabled grid portal that is wrapped around WS-PGRADE providing visualization and data management services to the scientific community by means of an easy-to-use graphical environment for accessing the full functionality of VisIVO Server. Complex workflows can be created and executed on a variety of infrastructures (e.g. clouds, desktop and service grids or supercomputers) to obtain comprehensive exploration and analysis of large-scale astrophysical datasets. The gateway offers role-based authorization modules and supports secure login.
Currently a number of main roles are implemented for
access as follows: guests, standard and advanced users and
administrators [
Both remote and local datasets can be uploaded - i.e. residing on a remote URL or locally on a user’s PC. For 8http://www.sci-bus.eu remote files the user must specify URL and optionally a user name and password for authentication. Depending upon the size of the datasets under consideration, remote uploads could last a long period. To resolve this situation VisIVO Gateway allows an off-line mode by means of a workflow submission so that users can issue upload commands and then simply close their current session - a follow up e-mail typically gives notification once the uploading operation is completed. The workflow employed for remote importing is illustrated in Figure 3. It allows generation of significant information for meta data exploration, e.g. statistics on data values, histogram calculation and plotting or a sample extraction of uploaded datasets. Such meta data is available through the Properties portlet and some can be modified by the user (e.g. renaming VBTs or related fields).
Fig. 3. Remote VisIVO Importer Workflow.
VisIVO Gateway automatically displays all applicable VisIVO Filter operations allowing input of the relevant parameters. Finally the VisIVO Viewer is employed for image display. A right click on any processed dataset in the Data Management portlet is used in conjunction with the View button to create user-prescribed VisIVO Viewer views. VisIVO Gateway further allows users to generate scientific movies. These can be useful not only to scientists to present and communicate their research results, but also to museums and science centres to introduce complex scientific concepts to general public audiences.
Users can create a Panoramic Movie by moving a camera along a motion path of 360o in azimuth and +/- 90o in elevation within the dataset’s domain. Customized Movies can be produced by intermediate snapshots specified as camera positions/orientations and the gateway generates a movie with a camera path containing these specified positions/orientations. Dynamic Movies can be created by interpolating several steps of a time evolution of a cosmological dataset. The user can browse a cosmological time evolution and choose two or more coherent datasets. The designed workflow will then produce the necessary number of intermediate VBTs by calculating particle positions and applying boundary conditions as necessary. This approach can be very useful, e.g. in revealing galaxy formation or observing large-scale structures such as galaxy clusters.
The creation of a movie represents a significant challenge
for the underlying computational resources as often hundreds
or thousands of high quality images must be produced. For this
reason Parameter Sweep (PS) workflows [
Once the data file is uploaded a sequence of simple actions is required to rapidly obtain meaningful visualizations. Typically various VisIVO Filter operations are performed, and The VisIVO Mobile application (see Fig. 5) allows smartphone devices to exploit VisIVO Gateway functionalities to access large-scale astrophysical datasets residing on a server repository for analysis and visual discovery. Through interactive widgets, customized visualizations (images or movies) can be generated and stored on the remote server. The application notifies users when requested visualizations are available for retrieving on their smartphones and allows sharing of data, images and movies via e-mail or by exploiting common social networks.
The current version of VisIVO Mobile is implemented
in Objective-C optimized for the Apple iPhone, iPod and
iPad, and, in the near future, it will be ported to other
popular smartphone devices. End users can login with the same
credentials as on the gateway and the application provides the
password coding in SHA cryptography exploiting the built-in
functionalities of the Liferay9 environment and querying the
remote database to verify access credentials. The configuration
and submission of workflows residing on the VisIVO Gateway
is performed by means of the gUSE Remote API [
9http://www.liferay.com 10http://curl.haxx.se/
The VisIVO Science Gateway is based on the collaborative and community oriented application development environment WS-PGRADE/gUSE. There is full integration in the portal framework Liferay which is highly customizable thanks to the adoption of portlet technology defined in the Java Specification Request 168 and 28611, and compatible to modern web applications. The implemented portlets are developed with the Java Vaadin web Framework12. This open source framework has been employed to implement server side Java Servlet based web applications using the full power and flexibility of Java without taking care of the client side since it compiles the Java source code to JavaScript which can then be run on any browser.
The current architecture of VisIVO Science Gateway has
a distributed configuration on different machines enhancing
the service performances as shown in Figure 6. The front-end
services contain WS-PGRADE and Liferay and the back-end
services include the gUSE components. The database server
resides on the back-end machine. The VisIVO community
of advanced users are enabled to create, change, invoke, and
monitor workflows accessing to all of the components of
WSPGRADE/gUSE, while standard users are provided with the
easy-to-use specific web based user interfaces described in
Section III-A including the gUSE Application Specific Module
(ASM) API [
The VisIVO Science Gateway currently exploits the
Cometa Consortium grid13. This infrastructure is distributed in
11http://jcp.org/en/jsr
12http://www.vaadin.com
13http://www.consorzio-cometa.it
seven sites of Sicily. All sites have the same hardware and
software configuration allowing high interoperability and realizing
an homogeneous environment. The computing infrastructure is
based on IBM Blade Centre each containing up to 14 IBM
LS21 blades interconnected with the low latency
Infiniband4X network, to provide High Performance Computing (HPC)
functionalities on the grid. There are currently about 2000
CPU cores and more than 200 TBs of disk storage space
available on this HPC e-Infrastructure. As reported in [
Building large workflows from scratch to address scientific communities can be time-consuming, as it is inherently a multi-disciplinary process. As an example, although astrophysicists might be able to appreciate the benefit to their work in using a workflow, they are less interested in the technical details for developing it, this is a task that is naturally associated with the developers (typically computer scientists). Manually monitoring the evolving structure of workflows, e.g. by email or written documentation, can be quite challenging. The plan is then to not only educate non computer science scientific communities in using workflows, but to also provide them with high level tools so that they can access the results of these workflows intuitively. Effective collaboration requires ways to facilitate exchange between different groups, in particular enabling sharing and realizing re-use and interoperability. The SHIWA project15 (SHaring Interoperable Workflows for large-scale scientific simulations on Available DCIs) provided solutions to facilitate sharing and exchanging of workflows between workflow systems and DCI resources through the SHIWA Simulation Platform (SSP) consisting of: SHIWA Repository16: A database where workflows and meta-data about workflows can be stored. The database is a central repository for users to discover and share workflows within and across their communities.
SHIWA Portal17: A web portal that is integrated with the SHIWA Repository and includes a workflow executor engine that can orchestrate various types of workflows on a number of computational grid/cloud platforms.
Through the SHIWA Portal one can define and run simulations on the SHIWA Virtual Organisation which is an einfrastructure that gathers computing and data resources from various DCIs, including the European Grid Infrastructure. The portal (via third party workflow engines) provides support for a number of commonly used academic workflow engines 14http://etics3.4dsoft.hu 15http://www.shiwa-workflow.eu 16http://shiwa-repo.cpc.wmin.ac.uk 17http://shiwa-portal2.cpc.wmin.ac.uk/liferay-portal-6.1.0 and it can be extended with other engines on demand. Such extensions translate between workflow languages and facilitate the nesting of workflows into larger workflows even when those are written in different languages and require different interpreters for execution. This functionality can enable scientific collaborations to share and offer workflows for reuse and execution. Shared workflows can be executed on-line, without installing any special client environment for downloading workflows.
A number of challenging workflows has been prototyped recently to support highly specialised scientific communities mainly in astrophysics. This section discusses our experiences with the visualisation-oriented workflows Muon Portal and LasMOG, and the simulation-oriented workflow FRANEC. The former are deployed for detecting nuclear threat materials (see V-A) and investigating large-scale modified gravity models (see V-B) respectively. The latter is exploited for carrying out stellar evolution simulations. These workflows will be supported in ER-flow18 so that they can be stored into the SHIWA workflow repository together with related meta-data, allowing investigation of their interoperability and dissemination across relevant communities through the SHIWA simulation platform.
Advanced users can exploit such workflows as templates for building new customized workflows to suit particular requirements of scientific communities, e.g. by modifying appropriately constituent building blocks customized LasMOG workflows can be generated. Standard users can then execute these workflows in an interactive and user-friendly way by means of the supplied portlets. Any user can submit jobs to the underlying DCIs without requiring a priori any specific technical expertise related to the particulars of the DCI configuration.
We are currently in the planning stages of developing a number of new visualisation-oriented workflows to be deployed for rapid discovery of supernova light curve anomalies19 and validation of models reconstructing the large scale structure of the universe2021. Furthermore two simulationoriented workflows are under development, the first one will be deployed for studying trajectories of interstellar comets passing through the Solar System and the second one will be focused on modelling the dynamical evolution of meteoroid streams. The vision is that, once a sufficient number of visualisation-oriented and simulation-oriented workflows has been developed, to analyse any similarities in depth towards developing templates for generating classes of workflows to address the needs of specialized scientific communities. The remaining of this section focuses on the Muon Portal, LasMOG and FRANEC workflows.
The deflection of muonic particles present in the secondary
cosmic radiation results from crossing high atomic number
materials (such as uranium or other fissile materials). This can
18http://www.erflow.eu
19http://supernovae.in2p3.fr/ guy/salt
20http://www.mpa-garching.mpg.de/gadget
21https://github.com/cmcbride/bgc utils
significantly improve on the success rate of current nuclear
threat detection methods which are based on X-ray
scanners [
We have developed a visualisation-oriented workflow
suitable for inspection of cargo containers carrying high atomic
number materials, by displaying tomographic images [
Further processing is then applied based on user-defined
thresholds, followed by conversion into data volumes using the
deflection angle field distribution by employing the 3D
Cloudin-Cell (CIC) [
Fig. 7. results.
Muon Portal processing: portlet interface, workflow and selected
The acceleration of the Universe is one of the most
challenging problems in cosmology. In the framework of general
relativity (GR), the acceleration originates from dark energy.
However, to explain the current acceleration of the Universe,
the required value of dark energy must be incredibly small.
Recently efforts have been made to construct models for
modified gravity (i.e. without introducing dark energy) as an
alternative to dark energy models [
Observing the large scale structure of the universe could in principle provide new test of GR on cosmic scales. This kind of test cannot be done without the help of simulations as the structure formation process is highly non-linear. Largescale simulations are thus performed for modified gravity models, e.g. from the Large Simulation for Modified Gravity (LaSMoG) consortium.
The workflow shown in Figure 8 implements a customised visualization for aiding analysis of modified GR simulations, more specifically inspecting datasets to discover anomalies by comparing appropriately with datasets coming from standard GR models. The main computational steps are summarised as follows:
Two datasets corresponding to snapshots of standard gravity (DS ) and modified gravity (DM ) model simulations are processed.
Sub-samples of the point distributions with a reduced number of points in the two datasets are generated. Then, for each of these sub-samples a panoramic movie is created (as shown in the resulting top image of Figure 8).
A point distribute operation is performed on DS and DM to create new volume datasets (VS and VM respectively) using a field distribution algorithm on a regular mesh.
A volume property on the same computational domain is distributed on a regular mesh producing a density field.
A new volume V is computed where each of its voxels shows a difference of values in the density between VS and VM . It is then filtered with a lower bound threshold and all the voxels satisfying the filters are saved in a text file for further analysis purposes (as shown in the resulting bottom image of Figure 8).
are performed:
Isosurface rendering of the density field to produce panoramic movies using different isovalues (as shown in the resulting bottom image of Figure 8); Ortho-slice rendering i.e. orthogonal slice planes through the volume dataset.
FRANEC is a state-of-the-art [
The simulations of stellar models produce simulation output files with a set of associated metadata. Such metadata are linked to all parameters concerning the numerical evolutionary code. In this way it is possible to store and easily search and retrieve the obtained data by many sets of stellar simulations, and furthermore get access to a large amount of homogeneous data such as tracks and isochrones computed by using FRANEC. The FRANEC workflow (see Figure 9) has a modular architecture making it easy to identify reusable modules for building other workflows. Modules can be differentiated on the basis of their functionality: 1) 2) 3)
EOS Computation module provides the Equation of State in tabular form. The input values are the Metallicity Z and the type of mixture (combination of chemical elements heavier than helium).
OPACITY Computation module produces a table of Opacity from pre-calculated tables. Given the Metallicity value Z and the type of mixture it obtains a new table of opacity which is interpolated from the pre-calculated ones.
FRANEC is the core module of the workflow. It produces the models of stellar evolution starting from the output of the two modules EOS and OPACITY and a set of input parameters given by the user to perform the evolution: the mass (in Solar Units) of the structure, the mass fraction of the initial helium, the mass fraction of the heavy elements abundance, the efficiency of superadibatic convection, the mass loss , the core convective overshooting during the Hburning phase , the diffusion index and the evolutionary stage index . It produces a set of parameter values varying in relation to time, quantities varying in relation to the radius of the model, the chemical composition of the core (vs. time), surface chemicals (vs. time), and energy resolution flows(vs. time).
Output Post-Processing module consists of the following jobs:
Traditionally the common practice among astronomers for data exploration tools was to employ small, individually created and executed applications. This scenario is not applicable to modern large-scale datasets. Modular web applications for data analysis and visual discovery making effective usage of modern e-infrastructures can be instrumental in reaching out astrophysical communities and aiding them in new scientific discoveries.
A workflow-oriented gateway allows scientists to share their analysis workflows and identify best practices for investigating their datasets. More importantly, they can automate workflows for repeated analysis with changed parameters, which in the past was a manual, slow and very error prone process. This way scientists can focus on core scientific discoveries rather than wasting time on data analysis on dealing with inadequate resources.
VisIVO Gateway provides a web based portal for setting up, running and evaluating visualizations in astrophysics for large-scale datasets exploiting DCIs resources. The gateway includes a data repository containing images and movies produced from imported datasets, as well as repositories of fundamental workflows, which can be used as templates for generating new workflows to be distributed by the users of the system.
We presented several portlets running in a Liferay portal environment together with a mobile application making the gateway accessible from modern mobile platforms. For a number of specialised astrophysical communities we have discussed workflows and the issues involved in developing them. The modularity achieved by subdividing workflows into a number of core tasks ensures re-usability and provides high flexibility. End users do not need to be aware of set-up options or be aware of the computing infrastructure operating behind the scenes.
We envisage building a specialized repository of astrophysics workflows core modules to share them among communities using the SHIWA platform. Our vision for these is to be used not only by astrophysical communities but to also be potentially exploited within other scientific contexts. This activity will also be instrumental in future work for creating an Astro-Gateway Federation establishing a network of Science Gateways to benefit astrophysical communities sharing tools and services, data, repositories, workflows and computing infrastructures.
The research leading to these results has received funding from the European Commission’s Seventh Framework Programme (FP7/2007-2013) under grant agreement no 283481 SCI-BUS (SCIentific gateway Based User Support) and the FP7 project under contract no 312579 ER-flow (Building an European Research Community through Interoperable Workflows and Data).