-This paper presents the latest developments on the VIALACTEA Science Gateway in the context of the FP7 VIALACTEA project. This science gateway operates as a central workbench for the VIALACTEA community in order to allow astronomers to process the new-generation (from Infrared to Radio) surveys of the Galactic Plane to build and deliver a quantitative 3D model of our Milky Way Galaxy. The final model will be used as a template for external galaxies to study star formation across the cosmic time. The adopted AGILE software development process allowed to fulfill the community needs in terms of required workflows and underlying resources monitoring. The scientific requirements arose during the process highlighted the needs for easy parameter setting, fully embarrassingly parallel computations and large-scale input dataset processing. Therefore the science gateway based on the WS-PGRADE/gUSE framework has been able to fulfill the requirements mainly exploiting the parameter sweep paradigm and parallel jobs execution of the workflow management system. Moving from the development to the production environment an efficient resource monitoring system has been implemented to easily analyse and debug sources of failure due to workflows computations. The results of the resource monitoring system are exploitable not only for IT experts administrators and workflow developers but also for the final users of the gateway. The affiliation to the STARnet Gateway Federation ensures the sustainability of the presented products after the end of the project, allowing the usage of VIALACTEA Science Gateway to all the stakeholders and not only to the community members.
The Milky Way is a complex ecosystem where a cyclical transformation process brings diffuse baryonic matter into dense unstable condensations to form stars, that produce radiant energy for billions of years before releasing chemically enriched material back into the InterStellar Medium in their final stages of evolution. Although considerable progress has been made in the last two decades in the understanding of the evolution of isolated dense molecular clumps toward the onset of gravitational collapse and the formation of stars and planetary systems, a lot remains still hidden.
The aim of the European FP7 VIALACTEA project is to exploit the combination of all new-generation surveys of the Galactic Plane to build and deliver a galaxy scale predictive model for star formation of the Milky Way. This model will be used as a template for external galaxies and studies of star formation across the cosmic time. Usually the essential steps necessary to unveil the inner workings of the galaxy as a star formation engine (such as the extraction of dust compact condensations or robust reconstruction of the spectral energy distribution of objects in star-forming regions) are often carried out manually by the astronomer, and necessarily over a limited number of galactic sources or very restricted regions.
Therefore scientists required new technological solutions able to deal with the growing data size and quantity coming from new-generation surveys (from Infrared to Radio wavelength). Moving to the Big Data era, allows to overcome the current challenges pushing the envelope of the current state of the art both from technological and scientific point of view. The extraction of the meaningful informations contained in the available data required an entirely new approach (the new paradigm of “data driven scientific discovery”) which resulted in a novel framework based on advanced visual analytics techniques, data mining methodologies, machine learning paradigms and Virtual Observatory (VO) based data representation and retrieval standards. All the underlying pipelines required by this framework (e.g. knowledge base catalogue creation, maps making for visual analytic) are available through the VIALACTEA Science Gateway.
The gateway (described in Section III) is based on the
WSPGRADE/gUSE [
This paper presents the latest developments on the VIALACTEA Science Gateway including the workflows designed for the community and the resource monitoring system. The workflows (see Section III-B) are mainly focused for performing intensive computations: map making, i.e. the formation of sky images from the instruments data; data mining to obtain band-merged catalogues relating sources with associated counterparts at different wavelengths; filamentary structure detection and extraction from images.
Due to the diverse variety of software and computing capabilities required by the workflows, a novel monitoring system has been developed within the gateway to test the status of the whole system. The monitoring covers different levels of tests (see Section IV) checking the gateway interoperability with the computing infrastructures and the workflow submission and execution processes. These tests are performed periodically and the resulting reports are published on the gateway so that also final users are aware of any failure of the system avoiding waste of time in debugging their work. Furthermore e-mail alerts are sent on any failure to the infrastructures administrators to promptly fix the problem.
ARCHITECTURE
In order to deliver a model of our galaxy with quantitative star formation laws, it is necessary to reveal and analyse throughout the galaxy the dense filamentary clouds where starforming clumps are found. These clumps are found in very different environments and in different evolutionary stages and their properties are characterized through detailed modelling of their Spectral Energy Distribution. Their exact location is determined using the most up to date distance estimators and all these pieces need to be assembled to get a new view of our Galaxy.
The Galactic distribution of Star Formation Rate (stellar mass produced per unit time) and Efficiency (stellar mass produced per unit mass of available dense gas) can be quantitatively related to the variety of physical agents that drive star formation in the Galaxy. The timely exploitation of the huge amount of data available requires new technological solutions able to overcome the current challenges pushing the envelope of the current state of the art both from technological and scientific point of view. Therefore it has been implemented a novel system based on advanced visual analytics techniques, data mining pipelines, VO-based standards and science gateway technologies. The implemented framework can be seen as an integrated workspace where the Visual Analytics Desktop Client, the Science Gateway embedding the Data Mining pipelines and the VIALACTEA Knowledge Base can be employed both as independent actors or as interacting components (see Figure 1).
The VIALACTEA Knowledge Base (VLKB) includes a combination of storage facilities, a Relational Data Base (RDB) server and web services on top of them. It allows easier searches and cross correlations between data and currently contains: 2D surveys, catalogue sources and related band merged information; structural informations such as filament structures or bubbles; and Radio Datacubes with search and cutout services. Data-mining and machine-learning pipelines are embedded within the Science Gateway as workflows and employed to carry out building of Spectral Energy Distributions, distance estimate and Evolutionary classification of hundreds of thousands of star forming objects on the Galactic Plane. All these produced results are then ingested to the VLKB. The Visual Analytics tool allows the interaction with the VIALACTEA data and to carry out complex tasks for multi-criteria data/metadata queries on the VLKB, subsample selection and further analysis processed over the science gateway, or real-time control of data fitting to theoretical models.
Due to the cross-domain scientists involved in the community (computer scientists, technologists and astronomers) AGILE software development approach has been adopted. This approach in fact promotes adaptive planning, evolutionary development, early delivery, and continuous improvement, and it encourages rapid and flexible response to change. Crossdisciplinary face-to-face meetings have been organized to promote an iterative, incremental and evolutionary framework based on several cycles of requirements and feedback sessions.
The science gateway is exploited by the scientists to
configure and run the VIALACTEA workflows implementing the
pipelines developed by the community (see Section III-B).
Furthermore the science gateway allows the Visual Analytic
tool to submit workflows through the usage of Remote API
[
The usage of the science gateway provides user-friendliness (intuitive user interface), efficiency (fast response time even for complex user requests), scalability (fast response time even for a large number of simultaneous user requests), robustness (keeps working under any circumstances and recovers gracefully from exceptions) and extensibility (easy to be extended with new interfaces and functionalities).
The VIALACTEA Science Gateway2 is based on a
customized version of WS-PGRADE/gUSE version 3.7 and is
affiliated with the STARnet Gateway Federation [
Parallelism: gUSE supports four levels of parallelism on workflow execution. The lowest level, or node-level parallelism, is where the application itself is prepared to utilize the benefits of multicore processors (multithread) or cluster systems (e.g. parallel execution using MPI). Besides this option, gUSE supports parallel execution of different jobs placed at different parallel branches of the workflow graph as the most intuitive and simple concurrent execution (branchlevel parallelism). A third level of parallelism covers the situation when one algorithm should be executed on a large parameter field, generally called parameter study or parameter sweep (PS) execution. The highest level of parallelism is where the execution of the same workflow is done in parallel. In fact, such a parallel execution of workflows can also be initiated by the user submitting the same workflow with different configurations.
DCI and Storage access: gUSE can access various DCIs
using the DCI Bridge [
Workflow Management System: the workflow creation
and parameter setting can be performed from the web interface
by importing the workflows from the repository or by creating
2VIALACTEA Science Gateway:http://via-lactea-sg00.iaps.inaf.it:8080
their new one using a web-based graph editor. The graph editor
has been recently improved (see [
The available VIALACTEA workflows are mainly designed for: map making, i.e. the production of high quality images from the raw instruments data; data mining to obtain bandmerged catalogues, whose entries consist of sources with associated counterparts at different wavelengths; filamentary structure detection and extraction from images. Specifically the following workflows have been identified.
MOSAIC: The MOSAIC workflow employs Unimap [
The inputs specify the tiles to be processed (longitude and wavelength) and the parameters of the Unimap application.
The workflow automatically imports the required data from the
Herschel infrared Galactic Plane Survey (Hi-GAL) [
The Instantiator job prepares the input tiles to be processed by the map maker embedded workflow, which computes each tile separately. The Generator job prepares the sub-tiles to be processed by the Map Maker job (Unimap). Finally, the output is given by the Collector job of the map maker embedded workflow and contains the maps in FITS (Flexible Image Transport System) file format.
PPMAP: The PPMAP workflow executes a Point Process
MAPping (PPMAP) [
The inputs specify the tiles to be processed and the parameters (one for each input tile) to be sent to the PPMAP application.
The workflow automatically imports the required data from the Hi-GAL Survey. The output is given by the collector job of the workflow and contains the maps as FITS file formats.
Q-FULLTREE: The Q-FULLTREE workflow performs
compact source identification through band-merging. The
application is based on the positional cross-match among sources
at different wavelengths. It is configured as a multi threaded
job splitting the single-band input catalogues into a
userchosen number of small sub-catalogues, with a user-selected
percentage of overlapping entries in order to avoid the loss
of merged sequences related to borderline entries. FT-Recap
framework. DCIs are though much more powerful but
somewhat less reliable than standalone desktop PCs due to their
inherent complexity (remote execution, data staging,
environment changes, etc); moreover, the running time of scientific
applications varies widely, can take minutes but even days or
weeks to complete. Any outage of the underlying DCIs might
cause breaking the flow of calculations, and in spite of built-in
failover mechanisms (re-submit jobs on failure), on error, it can
(FullTree-Recap), a post-processing application associated to be very difficult to localize without having information about
the band-merging workflow, is submitted to re-organize the proper behavior of the computing infrastructure. Debugging
output of the Q-FULLTREE in order to fulfil the Spectral complex workflows can be a very tedious and time consuming,
Energy Distribution visualization expectations of the Visual which require re-running the application several times, with
Analytic Desktop Client. The employed applications are coded slight modifications, added logging. Furthermore, sometimes
in Python and make internal use of the STILTS public library these errors are not even repeatable; temporary blackouts or
[
Filamentary Structure Detection: The workflow is de- and implemented in the VIALACTEA Science Gateway,
worksigned to perform filament extraction. The underlying appli- flow developers can make sure that all the related DCIs operate
cation [
IV. RESOURCE MONITORING Four levels of resource monitoring had been implemented:
Level 1 (PBS cluster infrastructure head node monitoring);
Continuous monitoring of the operational status (“health”) Level 2 (PBS cluster worker nodes environment monitoring); of the underlying distributed computing infrastructures (DCIs) Level 3 (Portal PBS cluster interoperability monitoring); and connected to the science gateway is of high importance, as they Level 4 (VIALACTEA, domain-specific, workflow operational serve as the actual platform performing scientific calculations monitoring). for the VIALACTEA Science Gateway, which functionality The lowest level, level 1 (called “PBS head nodes”), checks was yet missing from the base WS-PGRADE/gUSE portal that the DCI is indeed accessible from the portal (head node responds to ping, successful SSH connection can be established) and all the essential middleware commands (qsub, qstat, pbsnodes, etc.) operate as expected. Level 2 tests (called “PBS worker nodes”) scan through all worker nodes available in the DCI (they are all candidates of potential job execution) and checks, one-by-one, that the expected execution environment is available, such as enough disk space, necessary runtime environments and libraries (Java, IDL, Matlab, Python, etc.). Level 3 tests involve testing both the portal’s and the DCI’s functionality (“Portal-PBS interoperability”), which executes a probe workflow, composed of a single job. Level 4 tests (“Vialactea base workflows”) submit pre-created, domainspecific workflows having characteristics and requirements similar to other applications used in the customized portal, though with parameters resulting in less load compared to other full-fledged computations. Note that once tests on a certain level fail, tests at higher levels will fail too; thus the lowest level of failures help in locating the source of the problem as precisely as possible.
Figure 3 illustrates which main components of the system are covered by the different levels of tests (the higher the level, the more components are covered by the monitoring test).
For the different levels of monitoring activities different frequency can be specified, i.e. how often and at what time they are to be executed. For example, in the current VIALACTEA portal, level 1 tests are set to be executed every 3 hours; level 4 tests run once a day, at midnight. It makes possible to tune and schedule the load caused by monitoring system itself to avoid performance degradation might be experienced during normal use of the portal as much as possible. All results are recorded, so operational status of each resource can be traced back for the specified period of time (30 days, by default).
Also, for error events e-mail alerting can be requested for any number of e-mail addresses (primarily, system administrator is notified).
Monitoring data can be viewed by any user of the portal; changes to settings are however allowed by the portal administrators only. Monitoring results are summarized and visualized in the form of tables and charts on a web interface. Figure 4 shows level 2 resource monitoring results. The table (on the top of figure 4) shows the latest results and the frequency of these tests (6 hours). PASSED 10/10 means all tests (free disk space, Java, Matlab, Python, IDL) had passed on all 10 worker nodes available in cluster “muoni-server-02.oact.inaf.it”. The chart (on the bottom of figure 4) shows test results of the last 30 days, indicating outage on dates 12–16, 17, and 19 February; the DCI worked properly at other times.
As a result of the introduced monitoring service, in practice, gateway users could now verify, prior to running their workflows, whether the infrastructure of their choice is indeed available, operational, responsive, and, if not, they still had the possibility to choose another DCI. Also, on error, they could check past records to clarify whether the failure was due to infrastructure problems thus avoiding costly debugging.
Administrators were always notified about DCI outages on time, so they could fix issues as soon as possible and inform portal users about the incidents and expected time of recovery, respectively. Unwanted side effects of software configuration changes in clusters were also detected by the monitoring tool automatically (e.g., corrupted Matlab, IDL paths in worker nodes). Finally, due to historical data, it turned out that the capacity of one PBS cluster was insufficient; response times were at an acceptable level at weekends only, at very low load. The given infrastructure was then re-installed with more processors and more worker nodes to fulfill users’ needs.
munity is facing, different science gateways and
workflow technologies are being exploited. Apart from the
WSPGRADE/gUSE framework that has been extensively
employed by the authors, see e.g. [
In [
The Kepler7 scientific workflow system [
Finally, to our knowledge, none of the above solutions included a resource monitoring system able to check the status of the overall gateway interacting components, including the required runtime, as required by the VIALACTEA community.
There exist several resource monitoring tools available,
such as Ganglia[
In this paper we have introduced a new framework that allow astronomers to process the new-generation surveys of the Galactic Plane to build and deliver a quantitative model of Milky Way Galaxy. The presented science gateway operates as a central workbench for the VIALACTEA community allowing to deal with the growing data size and quantity coming from new-generation surveys. The extraction of the meaningful informations contained in the available data required an entirely new approach (the new paradigm of data driven scientific discovery) which resulted in a novel framework based on advanced visual analytics techniques, data mining methodologies, machine learning paradigms and Virtual Observatory based data representation and retrieval standards.
The focus of the presented workflow applications is on map making, i.e. the formation of sky images from the instruments data; data mining to obtain band-merged catalogues relating galactic sources with associated counterparts at different wavelengths; and filamentary structure detection and extraction from sky images. Furthermore we have highlighted how the usage of WS-PGRADE/gUSE framework have been able to fulfil the project requirements thanks to its key features: userfriendliness, efficiency, scalability, robustness and extensibility.
8Nagios: http://www.nagios.org 9Zabbix:http://www.zabbix.com 10Prometheus:https://prometheus.io
This paper also described a novel resource surveillance component integrated into WS-PGRADE/gUSE portal capable of checking operational status of the employed computational infrastructures based on Portable Batch Systems (PBS). The monitoring covers different levels of tests checking the gateway interoperability with the computing infrastructures and the workflow submission and execution processes. These tests are performed periodically and the resulting reports are published on the gateway so that also final users are aware of any failure of the system avoiding waste of time in debugging their work.
Amongst the things deserving further studies is the evaluation of MetaBrokering service of WS-PGRADE/gUSE which is capable of distributing and balancing the load among different distributed computing infrastructures. This will be exploited for parameter sweep jobs, such as the map making computations, avoiding excessive load of one resource with respect to other having higher capacity.