-Climate and biodiversity systems are closely interlaced across a wide range of scales. To better understand the mutual interaction between climate change and biodiversity there is a strong need for multidisciplinary skills, tools and a large variety of heterogeneous, distributed data sources. In this regard, the EUBrazilCloudConnect project provides a usercentric research environment built on top of a federated cloud infrastructure across Europe and Brazil to serve scientific needs. One of the test cases implemented in this project focuses on climate change and biodiversity research. The BioClimate is the Science Gateway of the use case. It aims at providing end-users with a highly integrated environment, addressing mainly data analytics requirements. This paper presents a complete overview about BioClimate and the scientific environment delivered to the user community at the end of the project.
Climate and biodiversity systems are closely interlaced across a wide range of scales. In order to predict the effects of climate change on the biodiversity system, which is essential towards sustainable landscape and eco-services management, there is a need to further investigate the interaction between the climate system and biodiversity.
Direct measurements of climate and biodiversity are often difficult and time-consuming to obtain, instead it is common practice to use climate and biodiversity indicators. These interactions can be studied at various scales, ranging from microscopic scales, and at (genomic, taxonomic, ecosystem) scales of individual plant and animal species. A multi-scale and integrated approach is required to investigate the climatebiodiversity system as a whole. Presently, in this scenario, researchers and professionals are burdened by scattered data sources, wealth of analysis tools to master and implement, and computational limitations to upscale their analysis.
EUBrazilCloudConnect [
The Biodiversity and Climate Change use case (BioClimate) involves multiple heterogeneous data sources (e.g. SEBAL, LiDAR, CRU, CMIP5, speciesLink, GBIF, etc.) and several processing pipelines, integrated through the BioClimate Scientific Gateway. The gateway sits on top of the databases and enables near-real-time analysis of large volume datasets (from multi-GBs to multi-TBs scale depending on the specific data source) through the Parallel Data Analysis Service (PDAS). PDAS clusters are deployed on the site where the databases are stored providing the end-user with a high-level, parallel, and server-side interface for scientific data analysis.
The design of the software infrastructure and the BioClimate Scientific Gateway for end-users facilitates joint research using data that is otherwise difficult to access or for which availability is fragmented and/or too large to process using traditional computational means. With regard to existing approaches and tools that are mainly client-side/desktop based, the use case delivers a well-integrated environment for climate change and biodiversity research with cloud-based infrastructure and server-side capabilities.
This work presents the BioClimate Scientific Gateway, the scientific challenges addressed and the implementation details. The remainder of this work is organised as it follows. Section II provides an overview of the BioClimate use case and its main goals. Section III provides a general description of the BioClimate Scientific Gateway architecture, whereas Section IV and Section V give, respectively, a detailed description of the graphic interface and the back-end. Finally, Section VI draws the main conclusions and describes the future activities. II. A BIODIVERSITY & CLIMATE CHANGE USE CASE The EUBrazilCloudConnect (EUBrazilCC) use case on climate change and biodiversity is a data-driven use case, aiming at better understanding the interactions between the biodiversity system and the climate system. This use case focuses on bringing together a wide variety of climate and biodiversity data and analysis tools into a user-friendly and web-based Science Gateway to provide an integrated approach of investigating climate and biodiversity across different temporal and spatial scales.
To address all these scientific challenges, the use case joins together heterogeneous data sources, on-premises cloud infrastructures, multiple data services, and a Science Gateway into a single, federated trans-Atlantic environment.
The Science Gateway provides access to historical
temperature and precipitation records, different climate model
scenarios with predictions of future temperature and precipitation,
Landsat [
The EUBrazilCC infrastructure provides the computing power needed to support data processing and analysis, the management of metadata to enable search and discovery as well as provenance management to address re-usability and reproducibility, both strongly relevant for scientific data environments. The BioClimate Scientific Gateway integrates in a web-based environment the data sources and the processing and analysis capabilities exploiting the project infrastructure. More specifically, the gateway has been designed to fulfil some key requirements:
Integration of heterogenous data sources. The gateway provides a unified interface to access and process satellite images (from Landsat), environmental data, future climate scenarios, biodiversity data like species distributions and LiDAR datasets related to some target areas. Furthermore, the gateway provides also metadata information describing these data sources.
analysis, several tools are integrated in the gateway to
allow: computation of 3D vegetation products based on
LiDAR data [
The software architecture of the use case is shown in Figure 1. The BioClimate Scientific Gateway represents the high-level user interface provided by the use case. It allows data access, analysis and visualisation over multiple, heterogeneous data sources, by exposing an integrated view of the data level. It supports several features, such as time-series and statistical analysis, data inspection, intercomparison and subsetting.
The elastic-job engine takes care of the execution of the
requests submitted through the gateway interface by translating
the requests in PDAS tasks and then properly scheduling
the jobs on the available resources. To guarantee scalability,
it elastically adapts to the analytics workload exploiting the
underlying cloud resources. The engine interacts with the
Infrastructure Manager (IM) [
A system catalog is used by both the front-end and the backend to store useful information regarding user management, experiment execution requests and results, PDAS cluster usage history and it also serves as a centralised data repository.
The PDAS, a core component of the Ophidia project [
The gateway also provides access to the BioClimate Clearing House, a database where the user can persistently store the results of the experiment run during a session and retrieve them through the search functionalities.
The lowest layer of the diagram comprises the several private clouds, running OpenNebula or OpenStack at the Infrastructure as a Service (IaaS) level, and the data sources, made available by the project partners or already available from national and international agencies, which are part of the infrastructure with a more static setup.
The data sources integrated through the gateway are reported in the following:
SEBAL datasets. These are an output of satellite images
series (Landsat) processed by the SEBAL [
LiDAR data. For the areas near Manaus in Brazil, where
hyper-spectral imagery is apparently absent, EUBrazil
Cloud Connect will leverage of the available LiDAR data
provided by EMBRAPA [
Biodiversity data sources. The speciesLink datasets [
(ESGF) [
gridded datasets (CRU TS v.3.23 [
Finally, security cuts across the whole architecture and
is taken into account at several levels. With regard to the
front-end, the security is implemented in terms of user
authentication. In order to avoid potential attacks that aim at
stealing passwords, the system employs a technique based on
salted password hashing, based on a Java implementation of a
Cryptographically Secure Pseudo-Random Number Generator,
called Password-Based Key Derivation Function 2 (PBKDF2)
[
At the elastic-job engine level, the PDAS terminal is used
to send requests to a PDAS server interface. It can exploit
the X509v3 digital certificates-based authentication and the
VOMS-based authorisation. Different levels of privileges are
defined to distinguish user roles locally at each PDAS server or
globally at the VOMS server. For this purpose, a GSI/VOMS
enabled interface, supporting both X.509 certificates and
VOMS-based authorisation and addressing the interoperability
with the EGI Fed Cloud environment [
In order to address portability of the system and the separation of concerns between the presentation layer and the business logic, the gateway has been implemented according to the Model-View-Controller pattern.
The presentation layer, running on the client side (i.e. a
browser), provides a rich user interface to submit the data
analysis tasks and visualise their results. It is implemented as
a JavaScript web application based on the ExtJS library [
The server side of the Science Gateway implements the
business logic to manage users, handle the requests and the
post-processing of the results and is based on Java and Apache
Struts2 framework [
To increase the performance and make the output visualisation faster, it has been decided to perform the heavier tasks, related to the post-processing of the outputs, on the server side and to present the ready-to-use result to the JavaScript library on the presentation layer.
Usability has been addressed by defining and implementing a set of pre-defined experiments regarding the different data sources and type of analysis. Each experiment defines a customisable template to perform data analytics tasks on climate and biodiversity data and requires a specific pipeline of operations, including subsetting, data reduction and mathematical/statistical functions.
The following subsections provide a description of the main views and interfaces made available by the gateway.
The ”Interactive analysis” panel allows a real-time, exploratory analysis of time series from the climate data available in the use case. In particular, it provides access to CRU historical data (temperature and precipitation variables) and future simulated data from the CMIP5 experiment (maximum and minimum temperatures from different climate models and scenarios).
As shown in Figure 2, the interface allows the selection of a dataset and a variable from the list of datasets/variables available and a point from the map. The bottom section of the Science Gateway displays the result of the analysis in terms of: (i) a chart with the time series and its trend line and (ii) a table with a comprehensive set of aggregated statistics.
The ”Compute” panel provides the features to define and submit complex experiments regarding the available data sources. For each experiment, a map for spatial selection and a form to set the input parameters is provided. The following experiments are defined:
provides information about interannual trends and statistical information of a specific SEBAL variable. The Science Gateway integrates data processed by the SEBAL algorithm and provides functionalities to analyse several variables produced by this algorithm (e.g. Enhanced Vegetation Index, Leaf Area Index, Normalized Difference Vegetation index, etc.). The interface allows both spatial and temporal selection.
Climate and SEBAL variables intercomparison allows the comparison of the behaviour of climate and SEBAL variables. In particular it supports analysis over the variables produced by the SEBAL algorithm and variables (precipitation and temperature) from historical climate data. From a scientific point of view, this experiment provides useful information about the relationship between climate and vegetation indices.
indicators computed on CMIP5 datasets belonging to
different climate models and future emission scenarios
(RCP4.5 and RCP8.5 [
Fig. 4. SEBAL Interannual analysis details interface
based on maximum and minimum temperature are
available for comparison (i.e. TXx, TNx, TXn, TNn [
evaluation of the statistical relationship between LiDAR products available through the gateway (e.g. DSM, DTM, CHM). In this case, a LiDAR tile can be selected from the map.
mation about relative height at different percentiles (25%, 50%, 66%, 75% and 90%) of the points in a LiDAR tile.
Once the computation of the experiment is completed, details about the experiment are available through the ”Experiment Details” section. Figure 4 displays the output produced by a SEBAL interannual experiment, whereas Figure 5 and Figure 6 display the output produced by a LiDAR intercomparison experiment and Climate-SEBAL intercomparison experiment respectively.
In particular, to better suit the experiment peculiarities, a specific detail view is provided for each experiment defined above. Hence, various gadgets organised in different fashions are used to display the results, among these are: line charts to display statistical values and trend lines; scatter plots to evaluate variable and indicators correlation; tables to show the results and statistical values; maps with the environmental scenario; images of the LiDAR products; and histograms of the point distribution.
Most of the information provided through the gadgets is also available for download in CSV, raster, GeoTIFF or PNG format, depending on the type of experiment run. Furthermore, metadata regarding the experiment is available in the same view.
The BioClimate Clearing House system allows users to store a relevant experiment, run during a session, for future analysis. A smart search feature is available to filter out the experiments saved into the Clearing House, based on: (i) spatial domain used for the experiment, (ii) experiment type and (iii) submission date.
The BioClimate Scientific Gateway includes two administrative interfaces that (i) allow managing users and their privileges and (ii) provide some information about the resources exploited dynamically by the gateway (i.e. PDAS cluster instances) as well as some statistics regarding the number of experiments executed in terms of their type and status. Through this dashboard (see Figure 7) it is possible to get some insights about the use of the system by the end-users. The charts mainly provide real-time monitoring information regarding the number of experiments running/pending and the status of the resources. In particular, a histogram shows the set of experiments and their distribution across the active PDAS instances for the last couple of minutes, whereas a pie chart shows the set of clusters currently running the experiments.
The elastic-job engine is designed to guarantee fast processing of the user requests by exploiting dynamically and elastically the federated cloud infrastructure. To meet scalability and performance requirements, the engine is implemented as multithreaded daemon, based on GNU C libraries, that exploits the PDAS capabilities to perform pipelines of analytics tasks.
Data-driven processing pipelines, based on PDAS operators, have been defined integrating different tools, services and data formats.
Management of the workload is performed exploiting a
smart scheduling algorithm, which provides dynamic job
scheduling over a set of queues. A job queue is associated to
each PDAS cluster running on the infrastructure. To
horizontally scale on the workload, a new PDAS instance is deployed
automatically on the private cloud resources when the number
of pending jobs on all the queues exceed a configurable
threshold. A more detailed description of the automated cloud
deployment (through the elastic-job engine) of the PDAS, as
well as of the queue policy adopted and its rationale, are out
of the scope of this paper and can be found in [
As mentioned before, the PDAS provides the capabilities to perform data analytics on large scientific datasets and includes a set of libraries able to deal with different data formats. In the EUBrazilCC project, the PDAS addresses scientific challenges related to the BioClimate use case and it is used for, both batch and interactive data analysis on NetCDF, LiDAR and remote sensing data.
All the outputs of the PDAS are stored in JSON format. This eases the integration of the results into web-contexts like the BioClimate Scientific Gateway and the parsing of the outputs from JavaScript and Python-based applications.
To address the data analytics requirements and support the
processing pipelines of the use case, several new features
and mathematical functionalities have been developed during
the project lifetime. In particular, regarding the interactive
analysis, an operator that allows data inspection and
on-thefly exploration of time series has been implemented, whereas
to run the batch experiments, processing pipelines made up
of several new operators and functions have been defined. To
integrate external tools, an operator to run scripts has also
been developed. Besides the previous extensions, the import
process has also been optimised to reduce the time required
to import large-scale datasets such as SEBAL output data.
Finally, to automate the deployment of PDAS instances in
the EUBrazilCC federated infrastructure, some cloud-based
scenarios, based on RADL files [
During the final validation phase of the EUBrazilCC project, the BioClimate use case was highly appreciated by the endusers, due to its ability to provide and deliver in the same environment tools, pipelines, analysis/visualisation features, and several data sources in an integrated manner.
User experience was good and the change of paradigm (process the data on the server-side) was evaluated as the key added value. Despite it requires a learning process, the BioClimate Scientific Gateway provides multiple views and analyses of the retrospective data gathered. High level user experience and usability have been two key requirements considered in the implementation phase.
A lot of interest was also raised by governmental & environmental agencies (both research & education) especially in Brazil. A set of follow-up actions will be put in place from the different partners even beyond the project lifetime (that was part of the project sustainability plan).
Finally, the impact on the user community was very high. The gateway was evaluated as seamlessly, flexibly and efficiently able to integrate a comprehensive and useful set of scientific data management tools to increase the mutual understanding between climate change and biodiversity.
This work was supported by the EU FP7 EUBrazilCC Project (Grant Agreement 614048), and CNPq/Brazil (Grant