-This paper describes the new EGI 'Applications on demand service', that the EGI collaboration specifically designed for individual researchers, small research teams and early phase research infrastructures that do not have dedicated computational and storage resources, online applications and science gateways to perform scientific data analysis. The described service is available at http://access.egi.eu and through a lightweight registration and user identity vetting process allows user-friendly access to a growing number of scientific applications and application hosting frameworks (science gateways, VREs) that are configured to use the dedicated pool of cloud computing and HTC clusters from EGI. The service operates as an open and extensible 'hub' for providers and einfrastructure user support teams who wish to federated and share applications and services with individual researchers, or small, fragmented communities, typically referred to as 'the long tail of science'.
SaaS; e-infrastructures; cloud;
EGI is an e-Infrastructure collaboration that provides advanced computing and data services for research and innovation. The collaboration operates a federated, publiclyfunded e-infrastructure that currently comprises more than 300 resource centers from Europe and beyond. Over the last decade this infrastructure was the enabler of digital research conducted by over 40,000 researchers through the whole spectrum of science from High-Energy Physics, to Earth Sciences, Life Sciences, Chemistry, Astrophysics, and Humanities.
EGI resource centers rely on the expertise of the ‘EGI
Foundation’, EGI’s coordinating institute located in
Amsterdam. EGI members – national compute/data centers (so
called NGIs) and Intergovernmental Research Organizations,
such as CERN – operate the compute, storage, application and
software services that comprise the ‘EGI infrastructure’. These
compute/storage providers allocate resources to scientific
communities via ‘Virtual Organisations’ (VOs), according to
institutional or regional/national priorities. A VO is the online
representation of a scientific user group whose members are
usually work in similar or related research areas, or are part of
the same scientific collaboration, for which reason they need
the same applications, software and underlying hardware
capabilities. Some of the biggest VOs of EGI represent
experiments of the Large Hadron Collider (ALICE, ATLAS,
CMS, LHCb) [
Since its start in 2010, EGI has well defined processes to
create and operate VOs for large, structured, international user
communities. These communities have well established and
long-term presence, moreover they are resourced well enough
to sustain skilled IT support teams who can instantiate and
operate VO services for the researchers. The most advanced
research infrastructures from the ESFRI Research
Infrastructure Roadmap [
At the same time individual researchers, small research teams and members of early phase research infrastructures often struggle to access applications, compute and data services in EGI. The tools and resource allocation policies that were designed for long-living, structured communities were recognized as unsuitable for these type of users because they typically involve: • • •
Obtaining and using an X.509 personal digital certificate from Certification Authority (CA) recognized by EGI.
Joining an existing VO that matches the requester’s research subject/goals, or setting up a new VO.
Integrating relevant scientific applications with the VO and fulfilling operational responsibilities, such as VO membership management, resource negotiations and community/application service monitoring. allocation specific
These individuals are often referred to as the ‘long tail of
science’ [
Recognizing the needs of this unique group, and trying to compensate for the loss of NGI services in this domain we designed and developed a new service within EGI: the ‘Applications on demand service’. The service was designed in late 2014, demonstrated in November 2015, reached Alpha release for early adopters in January 2017, and was opened for the general public in April 2017 as a Beta1 service. The service is available at http://access.egi.eu.
This paper introduces this new service offering and details the technologies and components that were developed and integrated to realize the service. The rest of the paper is organized as follows: Section II introduces the user and provider requirements that drove the system development. Section III describes the overall architecture and the main design considerations. Section IV provides details about the different system components that were developed/customized from existing EGI tools to establish the service. Section V in a table the already integrated applications and components of the service. In Section VI we describe the steps that an applications hosting framework provider has to follow to make his framework accessible within the EGI Applications on Demand service. In Section VII we present our plans for the extensions and improvement of the service, while in Section VIII we draw conclusions from the described work.
The design process for the new service started with a requirement collection and analysis that helped us to understand the needs/preferences of the long tail of science, as well as the constraints from the EGI resource/service providers’ side that we have to consider as we are aiming at a sustainable service. These needs and constraints are summaries in this section: A. Users’ needs: 1. Zero-barrier access: any user who carries out non-forprofit research should be able to get an account with a ‘start-up’ resource allocation on EGI to access scalable application services together with the underlying cloud/HTC/storage resources. 1 EGI Beta service: Service being developed while available for testing publicly. 2. 100% coverage: anyone with Internet access can become a user, no need to ask for personal travel to obtain special credentials (e.g. X.509 certificate). 3. Extendible and open: The service must be extendible with additional application services to support specialized scientific disciplines and users. The extension should be possible via open interfaces and protocols. 4. User-centric: Support for users should be available in as many EGI member countries as possible.
B. Service providers’ constaints: 1. Realistic: Define an architecture that’s implementable under the available effort levels. Reuse existing EGI technological building blocks as much as possible. 2. Secure: Provide as high quality user identity vetting, and tracking of user activities as possible (close to the existing solutions that are based on personal X.509 certificates). 3. Scalable: Be able to scale to 100s of compute/data/application providers. (The number and size of providers limits the users too). 4. Recognized: Have sufficient policies and tools that enforce the users to acknowledge the use of the service in scientific publications resulted from this use.
The architecture of the Applications on Demand service is presented in Figure 1. In the heart of the service there is a 'User Registration Portal' (URP). This is where new users enter to the service. The list of applications and underpinning services can be publicly browsed on the URP. Usage requires login. Login is possible with Google, Facebook and EGI Single-Sign-On accounts – All un-vetted accounts that are available for anyone with Internet access. Within the URP the user can setup a personal profile and can submit a resource access requests. The request includes an estimate of the compute and storage capacity that the user would like to use through the application(s) he/she would like to use in the service. The request can use the default capacity allocation (at the time of writing this is 1000 CPUh and 10 TB storage) or can be a customized request.
The access request is forwarded to the distributed service support team. A team member – from the applicant’s country or from the EGI Foundation responds to the request and vets the user identity, checks the capacity request. The vetting is done by assessing the validity of the information provided by the user in the profile, including links to departmental websites and scientific publications and projects relating to the requested e-infrastructure use. If needed, the support teams contacts the applicant in email or by phone. Routing the access requests to national EGI members not only allows us to conduct such conversation in local languages, but also to connect long tail users to national EGI support teams, reaching recognition and trust in national e-infrastructure initiatives. If the capacity request exceeds the default allocation then the estimate is double-checked and if needed negotiated resource providers of the service.
After the applicant’s resource access request is approved, the user profile is set to ‘active’ in the URP. This information is propagated to the connected application hosting environments (science gateways, Virtual Research Environments or similar). These environments operate as applications hosting frameworks, being equipped with a set of applications and user interfaces, manuals/guides best suited for those applications. For example, some gateways can be suited for workflow type applications, others for parameter study jobs, yet others for applications of a certain scientific discipline. The service is open for any application hosting framework (See Section VI for more details). Science gateway and Virtual Research Environments supported by the European Commission FP7 and H2020 work programmes are for example ideal candidates for integration.
Approved users can login to any of the connected
application hosting frameworks and use the embedded
applications. These applications are already configured to scale
to the distributed compute and storage systems that are
contributed to the service by EGI members. The resources are
made accessible for the applications in the form of
Infrastructure as a Service clouds joint into the ‘EGI Federated
Cloud’ [
The application use generates load on these resources and this is reported by the resources into the EGI accounting system. From this accounting system the user support teams can obtain statistics about individual users, as well as about application hosting frameworks, cloud/HTC/HPC sites. When a user exceeds the amount of compute/storage/network capacity that was allocated for him/her, the account can be suspended in the URP, blocking the user from further consumption through this service. The continued use requires a new justification from the user and a corresponding evaluation by the support team.
The service was created by customizing various existing
EGI components, and by developing a few new ones in order to
glue together the required elements into a single service. The
following customizations and developments were made:
• The User Registration Portal (URP) was developed by
CYFRONET as an extension of the e-GRANT EGI
resource allocation tool [
A resource pool was formed from EGI cloud and HTC
sites. The participating sites feel institutionally or
nationally responsible for supporting the long tail of
science, and therefore contribute with capacity into this
pool. The sites are joined together into an EGI Virtual
Organization called ‘vo.access.egi.eu’ [
Access to EGI resources requires short-living X.509
proxy certificates from the client side. The traditional
method is to generate such proxies either from a
longterm personal certificate, or from a robot certificate [
Application hosting frameworks that can provide
userfriendly interfaces to conduct scientific applications on
VO platform resources in cloud or HTC/HPC clusters.
The gateways use the identity federation of the URP to
allow access to approved users, and user the PUSP
mechanism to interact with cloud and HTC resources.
The service currently includes three applications hosting
frameworks: the WS-PGRADE [
The following Table 1 provides a summary of the components that are currently operated within the EGI Applications on Demand service.
• 17 applications from different scientific areas and tools have been already integrated into the existing applications hosting frameworks and are offered ‘as services’ to users: • Two policies were developed: A security policy for resource centers offering cloud/HTC/HPC/storage and an Acceptable Use Policy (AUP) for users.
implement by participating Resource Centers. The policy defines that offering resources in this service shall not negatively affect the security or change the security risk of any other VO. In particular, security incidents originating in the Applications on Demand service should not impact the IT Infrastructure in ways that are incompatible with the operational model of other VOs. This document also provides guidelines on the implementation of security procedures and controls to facilitate offering of the Service by Resource Centers and Science Gateways. The Guidelines contain normative information on how to implement the Policy.
responsibilities of the users – such as using the services only for activities that relate to the work that was described in the access request form; or what text to use in scientific publications to acknowledge the use of the infrastructure.
This section provides a short overview of the integration
steps that gateway/VRE providers must complete to contribute
to the service. Interested providers should consult with the
online manual for details [
There are two fundamental prerequisites of integrating an application hosting framework: (1) the framework must be a mature technology2 with demonstrated use within publicly funded science and (2) the framework must be already able to use cloud, HTC or data services from EGI, or be ready to do so (to enable the scale-out of the hosted applications).
As reported in the previous Sections, the URP provides the
identity federation that enables users to authenticate in any of
the connected application hosting frameworks with either
social credentials or EGI SSO accounts. In the current
implementation, this identity federation is implemented with
Unity [
For applications hosting frameworks based on Liferay
technology INFN-Catania has developed an OpenID Connect
module [
A second step of integration is using the userID provided by Unity to generate Per-User Sub-Proxies (PUSP) to secure user interaction with the EGI resources. For this, the provider can rely on the eToken server that was already presented in the previous Section.
Lastly, the technical integration is complete when the
service support team together with the framework provider 1.)
registers the framework in the EGI service registry (GOCDB)
to activate the availability and reliability monitoring for the
framework with the EGI ARGO service [
The ‘EGI Applications on Demand service’ was opened
for public use in April 2017. In the next few months we are
working on promoting the system to potential users, mainly
through the user support teams of EGI member states.
Promotion will be focused on countries where national support
is lacking or insufficient for the long tail of science.
In parallel with broadening the uptake of the new service we
are also planning to improve/expand the technical setup. This
work will cover the following areas:
1. Replacing PUSPs with short-term proxies generated by
the RC Auth service [
In this paper we presented a new EGI Applications on Demand service. The service is specifically designed to cater for the needs of individual researchers, small research collaborations and early-phase Research Infrastructures. The service provides easy to use environments for these user communities to request and access scalable scientific applications and application porting environments. The service includes also distributed computing and storage resources, and eliminates the need for users to form community-specific agreements with EGI providers, and to sustain skilled IT teams to operate VO services.
The new service does not replace the ‘traditional’, community/project specific and national VOs of EGI. Structured scientific collaborations and mature Research Infrastructures continue to require dedicated VOs in EGI because only those can host fully customized, communityspecific services and only they can offer dedicated compute and storage capacity at extreme scales. National ‘long tail VOs’ are also here to remain in those NGIs that have effort to support local users with national services. The EGI Foundation continues to negotiate and secure services for community-specific VO through Service Level and Operational Level Agreements (SLAs, OLAs).
The work presented in this paper has been supported by the EGI-Engage H2020 project (Grant number 654142). The authors would like to thank the contribution to the implementation of the described service to the following institutes: BIFI, CESGA, CNRS, CYFRONET, INFN-Catania, INFN-Bari, NIKHEF, SZTAKI, ULB-VUB, UPV.